Jul 24 13:07:30.666874 ip-10-0-136-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 13:07:30.666910 ip-10-0-136-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 13:07:30.666918 ip-10-0-136-14 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 13:07:30.667132 ip-10-0-136-14 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 13:07:40.720988 ip-10-0-136-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 13:07:40.721003 ip-10-0-136-14 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d40987c87a0746f6963ccc01fe15e023 --
Jul 24 13:09:40.666219 ip-10-0-136-14 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 13:09:41.032110 ip-10-0-136-14 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:09:41.032110 ip-10-0-136-14 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 13:09:41.032110 ip-10-0-136-14 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:09:41.032110 ip-10-0-136-14 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 13:09:41.032110 ip-10-0-136-14 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:09:41.033993 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.033867    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 13:09:41.038425 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038409    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.038425 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038424    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038428    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038431    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038434    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038437    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038440    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038443    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038445    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038448    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038456    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038459    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038462    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038465    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038467    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038470    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038473    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038475    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038478    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038480    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038483    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.038489 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038485    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038488    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038491    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038493    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038496    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038499    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038502    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038505    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038507    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038510    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038512    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038515    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038517    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038519    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038522    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038524    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038527    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038529    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038532    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038534    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.038981 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038536    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038539    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038542    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038545    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038548    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038550    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038552    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038555    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038557    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038561    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038565    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038568    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038570    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038573    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038576    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038578    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038581    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038583    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038586    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038588    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.039454 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038591    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038594    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038596    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038598    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038601    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038603    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038606    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038608    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038611    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038613    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038616    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038619    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038623    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038626    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038629    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038636    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038640    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038643    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038646    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.039958 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038648    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038651    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038653    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038656    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038658    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038661    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038665    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038669    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038671    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038674    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038676    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038679    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038681    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038684    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038686    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038688    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038691    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038694    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.040727 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.038696    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041184    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041203    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041211    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041214    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041217    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041220    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041223    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041225    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041253    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.041364 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041292    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041396    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041403    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041408    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041412    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041417    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041421    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041425    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041429    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041434    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041438    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041442    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041445    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041450    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041454    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041458    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041463    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041467    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041471    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041476    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.041606 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041480    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041484    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041487    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041492    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041496    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041500    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041504    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041507    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041511    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041515    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041520    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041523    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041532    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041536    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041543    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041547    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041551    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041558    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041567    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041570    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.042132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041573    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041576    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041578    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041581    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041584    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041586    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041589    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041592    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041594    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041597    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041599    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041602    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041604    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041607    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041610    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041612    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041615    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041618    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041620    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.042611 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041623    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041625    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041630    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041633    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041636    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041638    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041641    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041646    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041648    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041651    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041653    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041656    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041659    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041661    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041664    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041666    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041670    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041673    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041676    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041678    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.043098 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041681    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041683    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041686    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041688    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041691    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041693    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041697    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041700    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041703    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041707    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.041709    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042102    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042109    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042112    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042115    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042117    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042119    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042122    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042125    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.043605 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042129    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042132    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042134    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042137    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042140    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042143    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042146    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042148    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042151    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042154    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042157    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042159    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042161    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042164    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042166    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042169    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042171    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042174    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042176    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.044079 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042180    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042183    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042186    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042189    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042192    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042195    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042197    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042200    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042202    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042204    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042207    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042210    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042212    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042216    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042218    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042221    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042223    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042225    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042228    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.044544 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042230    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042233    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042235    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042238    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042240    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042243    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042245    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042247    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042250    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042252    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042254    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042257    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042259    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042262    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042264    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042267    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042269    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042271    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042274    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042276    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.045068 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042278    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042281    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042283    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042286    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042289    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042293    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042296    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042299    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042302    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042305    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042308    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042310    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042313    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042316    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042320    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042323    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042327    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042329    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042332    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.045543 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042334    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042336    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042339    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042341    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042344    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042346    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042349    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042351    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042353    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042356    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042358    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042361    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042363    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.042366    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042439    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042449    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042460    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042466    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042472    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042477    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042483    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 13:09:41.046020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042487    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042490    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042493    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042496    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042500    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042502    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042505    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042508    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042510    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042514    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042517    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042521    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042523    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042526    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042529    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042532    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042536    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042538    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042541    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042544    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042547    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042549    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042552    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042555    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 13:09:41.046559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042558    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042565    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042568    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042570    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042573    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042576    2556 flags.go:64] FLAG: --enable-server="true"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042582    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042586    2556 flags.go:64] FLAG: --event-burst="100"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042589    2556 flags.go:64] FLAG: --event-qps="50"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042591    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042594    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042597    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042600    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042603    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042606    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042609    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042612    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042615    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042618    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042621    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042624    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042627    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042630    2556 flags.go:64] FLAG: --feature-gates=""
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042633    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042636    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 13:09:41.047158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042638    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042641    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042644    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042647    2556 flags.go:64] FLAG: --help="false"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042649    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042652    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042655    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042658    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042661    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042664    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042667    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042670    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042674    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042678    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042681    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042684    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042687    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042690    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042692    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042695    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042697    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042700    2556 flags.go:64] FLAG: --lock-file=""
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042703    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042706    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 13:09:41.047770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042709    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042714    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042717    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042720    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042722    2556 flags.go:64] FLAG: --logging-format="text"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042725    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042729    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042731    2556 flags.go:64] FLAG: --manifest-url=""
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042734    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042738    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042740    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042744    2556 flags.go:64] FLAG: --max-pods="110"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042747    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042750    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042752    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042755    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042758    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042761    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042764    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042770    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042774    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042778    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042781    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 24 13:09:41.048370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042784    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042789    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042792    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042795    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042797    2556 flags.go:64] FLAG: --port="10250"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042800    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042803    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-093e790736023601e"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042806    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042808    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042811    2556 flags.go:64] FLAG: --register-node="true"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042814    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042817    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042821    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042824    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042826    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042830    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042832    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042835    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042838    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042840    2556 flags.go:64] FLAG: --runonce="false"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042843    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042846    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042849    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042851    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042854    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042857    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 13:09:41.048976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042860    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042863    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042865    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042869    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042871    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042875    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042878    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042881    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042883    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042889    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042891    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042894    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042898    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042900    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042903    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042906    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042908    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042911    2556 flags.go:64] FLAG: --v="2"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042915    2556 flags.go:64] FLAG: --version="false"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042919    2556 flags.go:64] FLAG: --vmodule=""
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042922    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.042925    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043051    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043056    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.049661 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043059    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043063    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043066    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043068    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043071    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043073    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043075    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043078    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043080    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043082    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043085    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043089    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043091    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043095    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043098    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043100    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043103    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043105    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043108    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043110    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.050271 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043113    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043115    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043118    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043120    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043123    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043125    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043128    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043131    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043133    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043136    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043138    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043141    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043143    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043146    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043148    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043151    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043153    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043156    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043158    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043160    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.050757 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043163    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043165    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043168    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043173    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043176    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043180    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043182    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043185    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043187    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043190    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043192    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043194    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043197    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043199    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043202    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043204    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043207    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043209    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.051275 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043211    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043214    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043216    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043219    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043222    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043224    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043226    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043229    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043231    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043234    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043236    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043238    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043241    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043243    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043246    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043248    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043250    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043254    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043257    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043261    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.051700 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043263    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043266    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043268    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043270    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043273    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043275    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043277    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043280    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043282    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043284    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043288    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043291    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043294    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043297    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043299    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043302    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043305    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043308    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.052202 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.043310    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.043316    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.050301    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.050319    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.051086    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.051097    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051157    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051163    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051166    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051169    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051172    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051174    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051177    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051179    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.052630 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051182    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051184    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051187    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051190    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051192    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051195    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051197    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051200    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051202    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051205    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051207    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051210    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051212    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051215    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051217    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051219    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051222    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051224    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051227    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051229    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.053004 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051232    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051239    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051242    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051245    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051249    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051253    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051256    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051259    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051261    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051264    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051267    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051269    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051272    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051276    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051279    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051282    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051285    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051288    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051291    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.053561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051294    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051296    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051299    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051302    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051304    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051307    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051310    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051312    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051315    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051317    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051320    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051322    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051325    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051327    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051331    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051334    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051337    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051339    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051342    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051345    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.054048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051347    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051349    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051352    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051354    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051356    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051359    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051361    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051363    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051366    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051369    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051373    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051375    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051379    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051381    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051384    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051386    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051389    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051391    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051394    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051396    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.054531 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051398    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051401    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051403    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051406    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051408    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051411    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051414    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051416    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051420    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051422    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051424    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051427    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.051432    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051568    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051573    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:09:41.055174 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051575    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051578    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051581    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051584    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051586    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051588    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051591    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051593    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051596    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051598    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051601    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051603    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051605    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051608    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051610    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051613    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051616    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051618    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051621    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051623    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:09:41.055538 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051626    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051628    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051632    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051635    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051637    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051640    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051643    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051646    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051650    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051653    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051656    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051658    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051661    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051663    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051666    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051668    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051670    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051673    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051675    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:09:41.056030 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051678    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051680    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051682    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051685    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051687    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051690    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051692    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051695    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051697    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051699    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051702    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051705    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051707    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051710    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051712    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051716    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051719    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051722    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051724    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051726    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:09:41.056477 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051730    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051732    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051735    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051737    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051740    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051742    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051745    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051747    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051750    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051752    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051754    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051757    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051759    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051762    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051764    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051766    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051770    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051773    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051776    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051779    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:09:41.056972 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051782    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051784    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051787    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051791    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051794    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051797    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051799    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051802    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051804    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051807    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051809    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051812    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051814    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051817    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051820    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051822    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051824    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:09:41.057469 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:41.051827    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.051831    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.051974    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.054085    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.055509    2556 server.go:1019] "Starting client certificate rotation"
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.055610    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:09:41.057878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.055653    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:09:41.058150 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.058137    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:09:41.061689 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.061669    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:09:41.071874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.071846    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 24 13:09:41.074266 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.074251    2556 log.go:25] "Validated CRI v1 image API"
Jul 24 13:09:41.075109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.075092    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 13:09:41.079435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.079416    2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bb4cb951-7bc7-4c2e-8e86-ff0fdd3afc8e:/dev/nvme0n1p3 e6e7ceec-c4d8-44fc-ab84-f065a4e52c55:/dev/nvme0n1p4]
Jul 24 13:09:41.079501 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.079435    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 13:09:41.082926 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.082905    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:09:41.085199 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.085092    2556 manager.go:216] Machine: {Timestamp:2026-07-24 13:09:41.083973553 +0000 UTC m=+0.308456871 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103197 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2992964c1a28a91d1e47a5214eff4c SystemUUID:ec299296-4c1a-28a9-1d1e-47a5214eff4c BootID:d40987c8-7a07-46f6-963c-cc01fe15e023 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1d:0b:14:76:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1d:0b:14:76:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:94:91:5a:1d:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 13:09:41.085199 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.085195    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 13:09:41.085304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.085274    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 13:09:41.087050 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.087025    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 13:09:41.087182 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.087053    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 13:09:41.087922 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.087913    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 13:09:41.087970 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.087924    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 13:09:41.087970 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.087951    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:09:41.088888 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.088869    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:09:41.090006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.089997    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 13:09:41.091757 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091746    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:09:41.091887 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091877    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 13:09:41.091942 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091908    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 13:09:41.091942 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091923    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 13:09:41.091999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091949    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 13:09:41.091999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091959    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 24 13:09:41.091999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.091980    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 13:09:41.092363 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.092351    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:09:41.092406 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.092370    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:09:41.095814 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.095797    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 13:09:41.098406 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.098390    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 13:09:41.098498 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.098429    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 13:09:41.100179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100165    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100186    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100195    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100205    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100217    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100225    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100236    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100244    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100255    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 13:09:41.100263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100263    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 13:09:41.100516 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100278    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 13:09:41.100516 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.100286    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 13:09:41.101136 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.101124    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 13:09:41.101204 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.101139    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 13:09:41.101513 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.101495    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxdrr"
Jul 24 13:09:41.103018 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.103000    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 13:09:41.103377 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.103354    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 13:09:41.103437 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.103354    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 13:09:41.104357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.104341    2556 server.go:1268] "Started kubelet"
Jul 24 13:09:41.104435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.104398    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 13:09:41.105160 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.105102    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 13:09:41.105237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.105184    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 13:09:41.105262 ip-10-0-136-14 systemd[1]: Started Kubernetes Kubelet.
Jul 24 13:09:41.107536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.107512    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 13:09:41.109008 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.108988    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 13:09:41.109235 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.109212    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxdrr"
Jul 24 13:09:41.110098 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.110071    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 13:09:41.118858 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.118843    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 13:09:41.118946 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.118844    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 13:09:41.119060 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.119035    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.119744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.119721    2556 factory.go:55] Registering systemd factory
Jul 24 13:09:41.119744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.119747    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 24 13:09:41.119915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.119900    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 13:09:41.119915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.119905    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 13:09:41.120072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.119943    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 13:09:41.120072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120028    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 13:09:41.120072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120038    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 13:09:41.120235 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120193    2556 factory.go:153] Registering CRI-O factory
Jul 24 13:09:41.120235 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120207    2556 factory.go:223] Registration of the crio container factory successfully
Jul 24 13:09:41.120324 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120260    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 13:09:41.120324 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120282    2556 factory.go:103] Registering Raw factory
Jul 24 13:09:41.120324 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120296    2556 manager.go:1196] Started watching for new ooms in manager
Jul 24 13:09:41.120746 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.120732    2556 manager.go:349] Starting recovery of all containers
Jul 24 13:09:41.121557 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.121537    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:41.123797 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.123777    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-14.ec2.internal\" not found" node="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.128555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.128428    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 13:09:41.131130 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.131111    2556 manager.go:354] Recovery completed
Jul 24 13:09:41.135149 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.135136    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.137504 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.137489    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.137577 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.137516    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.137577 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.137527    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.138109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.138097    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 13:09:41.138109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.138108    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 13:09:41.138228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.138113    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.138228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.138123    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:09:41.140228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.139551    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.140228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.139631    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.140228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.139647    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.140355 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.140274    2556 policy_none.go:49] "None policy: Start"
Jul 24 13:09:41.140355 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.140287    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 13:09:41.140355 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.140298    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 13:09:41.141471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.141460    2556 policy_none.go:47] "Start"
Jul 24 13:09:41.187661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.187523    2556 manager.go:337] "Starting Device Plugin manager"
Jul 24 13:09:41.187751 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.187715    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 13:09:41.187751 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.187729    2556 server.go:87] "Starting device plugin registration server"
Jul 24 13:09:41.187997 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.187984    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 13:09:41.188059 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.187999    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 13:09:41.188132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.188095    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 13:09:41.188190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.188178    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 13:09:41.188239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.188193    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 13:09:41.188746 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.188729    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 13:09:41.188815 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.188781    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.289177 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.289115    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.290236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.290155    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.290236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.290182    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.290236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.290194    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.290236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.290220    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.298476 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.298461    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.298547 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.298483    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-14.ec2.internal\": node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.313731 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.313705    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.334492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.334475    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 13:09:41.334572 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.334511    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 13:09:41.334572 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.334531    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 13:09:41.334572 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.334560    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 13:09:41.336854 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.336839    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:41.414360 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.414311    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.434765 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.434740    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"]
Jul 24 13:09:41.434821 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.434805    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.436258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.436245    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.436327 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.436272    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.436327 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.436282    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.437456 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.437441    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.437600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.437586    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.437688 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.437619    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.438131 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438118    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.438218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438133    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.438218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438147    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.438218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438157    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.438218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438162    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.438218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.438168    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.439357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.439342    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.439436 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.439373    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:09:41.440065 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.440044    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:09:41.440166 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.440074    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:09:41.440166 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.440085    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:09:41.460543 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.460514    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-14.ec2.internal\" not found" node="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.465010 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.464994    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-14.ec2.internal\" not found" node="ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.514408 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.514386    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.522445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.522426    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd517d74ad9c69ffc34f61061c12dc4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-14.ec2.internal\" (UID: \"fd517d74ad9c69ffc34f61061c12dc4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.522512 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.522449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.522512 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.522466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.614756 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.614681    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.623155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623131    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd517d74ad9c69ffc34f61061c12dc4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-14.ec2.internal\" (UID: \"fd517d74ad9c69ffc34f61061c12dc4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.623228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd517d74ad9c69ffc34f61061c12dc4c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-14.ec2.internal\" (UID: \"fd517d74ad9c69ffc34f61061c12dc4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.623228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.623320 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.623320 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623290    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.623320 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.623290    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b510b9e2c520fe00d8e8ab6156268a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal\" (UID: \"d4b510b9e2c520fe00d8e8ab6156268a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.715275 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.715239    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.763781 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.763760    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.769388 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:41.769370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:41.816221 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.816192    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:41.916764 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:41.916706    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.017312 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:42.017274    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.055122 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.055083    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 13:09:42.055560 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.055268    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 13:09:42.055560 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.055279    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 13:09:42.116082 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.116045    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 13:04:41 +0000 UTC" deadline="2028-05-10 02:17:09.623047276 +0000 UTC"
Jul 24 13:09:42.116082 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.116072    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15733h7m27.506977815s"
Jul 24 13:09:42.117365 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:42.117345    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.218283 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:42.218250    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.263151 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:42.263118    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd517d74ad9c69ffc34f61061c12dc4c.slice/crio-302ba642efc4f57bf1753be590b93c53e31d014198555849ce78a4dd45b6c9fe WatchSource:0}: Error finding container 302ba642efc4f57bf1753be590b93c53e31d014198555849ce78a4dd45b6c9fe: Status 404 returned error can't find the container with id 302ba642efc4f57bf1753be590b93c53e31d014198555849ce78a4dd45b6c9fe
Jul 24 13:09:42.268882 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.268868    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:09:42.319372 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:42.319339    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.337610 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.337569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal" event={"ID":"fd517d74ad9c69ffc34f61061c12dc4c","Type":"ContainerStarted","Data":"302ba642efc4f57bf1753be590b93c53e31d014198555849ce78a4dd45b6c9fe"}
Jul 24 13:09:42.421768 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:42.421706    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-14.ec2.internal\" not found"
Jul 24 13:09:42.464785 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.464765    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:42.515050 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.515029    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:42.519168 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.519146    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:42.532635 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.532617    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:09:42.534429 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.534415    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal"
Jul 24 13:09:42.553665 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:42.553635    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:09:42.561114 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:42.561089    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b510b9e2c520fe00d8e8ab6156268a.slice/crio-fd3c2c02fb0a23940329582ad3d13a4f54c152927f2914a308e1d2527648ceff WatchSource:0}: Error finding container fd3c2c02fb0a23940329582ad3d13a4f54c152927f2914a308e1d2527648ceff: Status 404 returned error can't find the container with id fd3c2c02fb0a23940329582ad3d13a4f54c152927f2914a308e1d2527648ceff
Jul 24 13:09:43.093380 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.093345    2556 apiserver.go:52] "Watching apiserver"
Jul 24 13:09:43.101289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.101261    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 13:09:43.102083 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.102055    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal","openshift-multus/multus-additional-cni-plugins-ngptk","openshift-multus/multus-wfz4v","openshift-dns/node-resolver-lwjjd","openshift-image-registry/node-ca-l4qff","openshift-multus/network-metrics-daemon-rsqdv","openshift-network-diagnostics/network-check-target-g75k9","openshift-network-operator/iptables-alerter-wxt6p","openshift-ovn-kubernetes/ovnkube-node-7hx9v","kube-system/global-pull-secret-syncer-5mp7r","kube-system/konnectivity-agent-k4z72","kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw","openshift-cluster-node-tuning-operator/tuned-dfdtk"]
Jul 24 13:09:43.103993 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.103976    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:43.104093 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.104044    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:43.105632 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.105609    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.106876 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.106856    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.108101 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.108072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.109672 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.109381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.110761    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.110884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.111004    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111071    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111191    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111207    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111292    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.111408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111297    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.111834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111471    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.111834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111769    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qtmgd\""
Jul 24 13:09:43.111911 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.111898    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.112085 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.112073    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7svfn\""
Jul 24 13:09:43.112224 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.112205    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.112426 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.112407    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q579w\""
Jul 24 13:09:43.112600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.112586    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.113077    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.113279    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z956h\""
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.113455    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.113659    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.114213    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.115669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.114366    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.116556 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.116298    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 13:09:43.116556 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.116414    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.116702 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.116564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117042    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117218    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwmw9\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117403    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117475    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117536    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117730    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fk6cv\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.117780    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.118058    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.118161    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.118310 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.118249    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:43.118838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.118492    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 13:09:43.118898 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.118871    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 13:09:43.118898 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.118876    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2w8tr\""
Jul 24 13:09:43.119566 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.119548    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 13:09:43.120499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.120468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.122344 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.122324    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 13:09:43.122564 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.122545    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.122709 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.122693    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 13:09:43.122911 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.122893    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d4szw\""
Jul 24 13:09:43.129541 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.129521    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:09:43.129806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.129792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nw4tq\""
Jul 24 13:09:43.129996 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.129968    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 13:09:43.130102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.130075    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 13:09:43.131653 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131636    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-systemd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.131738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.131807 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131748    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-netns\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.131807 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-conf\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.131909 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-var-lib-kubelet\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.131909 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-host\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.132032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131906    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f5w\" (UniqueName: \"kubernetes.io/projected/27028e10-d16d-4940-8fb7-f76768ac35df-kube-api-access-h9f5w\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.132032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.131988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-cnibin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.132032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jc4\" (UniqueName: \"kubernetes.io/projected/76f64980-04b2-4e0b-8615-12c21a3f335b-kube-api-access-x8jc4\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.132228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132062    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.132228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132125    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0053d784-2bcf-4674-a6fa-2301aa65cab3-serviceca\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.132228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132154    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0053d784-2bcf-4674-a6fa-2301aa65cab3-host\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.132228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-daemon-config\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132256    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-etc-kubernetes\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132275    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkt5\" (UniqueName: \"kubernetes.io/projected/0053d784-2bcf-4674-a6fa-2301aa65cab3-kube-api-access-dfkt5\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132323    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-netd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.132419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-script-lib\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhqd\" (UniqueName: \"kubernetes.io/projected/150f7de7-ecae-42c2-8942-dde8cc5c3203-kube-api-access-4lhqd\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-os-release\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgxr\" (UniqueName: \"kubernetes.io/projected/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-kube-api-access-8fgxr\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132534    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-slash\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-kubernetes\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-tmp\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/672e4e80-f74f-4e3c-a20f-3167fe4970b5-hosts-file\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.132708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-ovn\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-bin\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132859    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-config\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-env-overrides\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.132925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.133073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672e4e80-f74f-4e3c-a20f-3167fe4970b5-tmp-dir\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.133389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvsgh\" (UniqueName: \"kubernetes.io/projected/672e4e80-f74f-4e3c-a20f-3167fe4970b5-kube-api-access-qvsgh\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.133389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.133389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-socket-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.133389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133310    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.133389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133372    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-var-lib-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-etc-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-bin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.133625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-multus\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.133625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:43.133856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-sys-fs\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.133856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz8d8\" (UniqueName: \"kubernetes.io/projected/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kube-api-access-qz8d8\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.133856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovn-node-metrics-cert\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.133856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrrp\" (UniqueName: \"kubernetes.io/projected/2d66e606-fc0c-4031-8314-7320e1ca9c74-kube-api-access-qzrrp\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133883    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-conf-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.133960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-multus-certs\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-netns\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-node-log\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134094    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0445648-3793-4fc2-80b6-1f0b2733536c-agent-certs\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.134128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-system-cni-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-log-socket\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-modprobe-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-sys\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-kubelet-config\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.134444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-os-release\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-registration-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-lib-modules\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134516    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-dbus\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0445648-3793-4fc2-80b6-1f0b2733536c-konnectivity-ca\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-socket-dir-parent\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.134764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-kubelet\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.134674    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-device-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-kubelet\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-systemd-units\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135361    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-etc-tuned\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-system-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-cni-binary-copy\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135426    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-k8s-cni-cncf-io\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135442    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135448    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysconfig\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-systemd\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-run\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-hostroot\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.135581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.135551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cnibin\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.151742 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.151720    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fts96"
Jul 24 13:09:43.159303 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.159282    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fts96"
Jul 24 13:09:43.221107 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.221080    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 13:09:43.235700 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235677    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-tmp\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.235850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/672e4e80-f74f-4e3c-a20f-3167fe4970b5-hosts-file\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.235850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.235850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28f7t\" (UniqueName: \"kubernetes.io/projected/31bd44cc-1c81-46bd-9083-717dae54b5db-kube-api-access-28f7t\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.235850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235819    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-ovn\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.235850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235840    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-bin\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-config\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235880    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-ovn\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-bin\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-env-overrides\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235974    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.235837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/672e4e80-f74f-4e3c-a20f-3167fe4970b5-hosts-file\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31bd44cc-1c81-46bd-9083-717dae54b5db-iptables-alerter-script\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672e4e80-f74f-4e3c-a20f-3167fe4970b5-tmp-dir\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236056    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236094    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvsgh\" (UniqueName: \"kubernetes.io/projected/672e4e80-f74f-4e3c-a20f-3167fe4970b5-kube-api-access-qvsgh\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.236132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-socket-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-var-lib-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-env-overrides\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.236591    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.236686    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:09:43.736667174 +0000 UTC m=+2.961150493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-etc-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236767    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236793    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-bin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-multus\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.236834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672e4e80-f74f-4e3c-a20f-3167fe4970b5-tmp-dir\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-multus\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-etc-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236917    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-sys-fs\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-cni-bin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.236976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz8d8\" (UniqueName: \"kubernetes.io/projected/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kube-api-access-qz8d8\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237014    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovn-node-metrics-cert\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-var-lib-openvswitch\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrrp\" (UniqueName: \"kubernetes.io/projected/2d66e606-fc0c-4031-8314-7320e1ca9c74-kube-api-access-qzrrp\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-socket-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-conf-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-multus-certs\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-netns\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.237407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-node-log\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0445648-3793-4fc2-80b6-1f0b2733536c-agent-certs\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-system-cni-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-multus-certs\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-log-socket\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237314    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-conf-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-modprobe-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237353    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-sys\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-sys-fs\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-kubelet-config\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237375    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-system-cni-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-os-release\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31bd44cc-1c81-46bd-9083-717dae54b5db-host-slash\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.238228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-run-netns\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237460    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-node-log\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-registration-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-os-release\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-sys\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-lib-modules\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237548    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-dbus\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0445648-3793-4fc2-80b6-1f0b2733536c-konnectivity-ca\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-lib-modules\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-modprobe-d\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-log-socket\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-kubelet-config\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-socket-dir-parent\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237677    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-registration-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-kubelet\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.239043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237723    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-device-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-socket-dir-parent\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-kubelet\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237782    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ec6639f-9950-4bd0-85d5-a3542940b6aa-dbus\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-var-lib-kubelet\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b847b1ff-f1ba-47ce-80c4-5ff306240e68-device-dir\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-systemd-units\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.237989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-etc-tuned\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238056    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-system-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-cni-binary-copy\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-k8s-cni-cncf-io\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238127    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysconfig\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0445648-3793-4fc2-80b6-1f0b2733536c-konnectivity-ca\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-config\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-systemd\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238190    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-systemd-units\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-run\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-k8s-cni-cncf-io\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.239833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysconfig\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-system-cni-dir\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238293    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-systemd\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238346    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-kubelet\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-hostroot\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238389    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cnibin\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-run\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-systemd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-hostroot\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-netns\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cnibin\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238487    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-conf\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-var-lib-kubelet\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-host\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238539    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-run-systemd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f5w\" (UniqueName: \"kubernetes.io/projected/27028e10-d16d-4940-8fb7-f76768ac35df-kube-api-access-h9f5w\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238584    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-cnibin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.240874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jc4\" (UniqueName: \"kubernetes.io/projected/76f64980-04b2-4e0b-8615-12c21a3f335b-kube-api-access-x8jc4\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0053d784-2bcf-4674-a6fa-2301aa65cab3-serviceca\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0053d784-2bcf-4674-a6fa-2301aa65cab3-host\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-cni-binary-copy\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-host-run-netns\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-var-lib-kubelet\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-host\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-daemon-config\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238796    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-cnibin\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239277    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/76f64980-04b2-4e0b-8615-12c21a3f335b-multus-daemon-config\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0053d784-2bcf-4674-a6fa-2301aa65cab3-host\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238656    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-sysctl-conf\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239360    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-etc-kubernetes\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.241740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.238825    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76f64980-04b2-4e0b-8615-12c21a3f335b-etc-kubernetes\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0053d784-2bcf-4674-a6fa-2301aa65cab3-serviceca\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkt5\" (UniqueName: \"kubernetes.io/projected/0053d784-2bcf-4674-a6fa-2301aa65cab3-kube-api-access-dfkt5\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-netd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-script-lib\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.239487    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhqd\" (UniqueName: \"kubernetes.io/projected/150f7de7-ecae-42c2-8942-dde8cc5c3203-kube-api-access-4lhqd\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.239539    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:09:43.739522154 +0000 UTC m=+2.964005461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-cni-netd\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-os-release\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239587    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgxr\" (UniqueName: \"kubernetes.io/projected/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-kube-api-access-8fgxr\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-slash\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-kubernetes\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-os-release\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27028e10-d16d-4940-8fb7-f76768ac35df-etc-kubernetes\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d66e606-fc0c-4031-8314-7320e1ca9c74-host-slash\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.239975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-tmp\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.242746 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.240140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovnkube-script-lib\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.242746 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.240218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0445648-3793-4fc2-80b6-1f0b2733536c-agent-certs\") pod \"konnectivity-agent-k4z72\" (UID: \"f0445648-3793-4fc2-80b6-1f0b2733536c\") " pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.242746 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.240881    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27028e10-d16d-4940-8fb7-f76768ac35df-etc-tuned\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.242746 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.242052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d66e606-fc0c-4031-8314-7320e1ca9c74-ovn-node-metrics-cert\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.247670 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.247648    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:43.247670 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.247671    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:43.247849 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.247681    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:43.247849 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.247728    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:09:43.747714121 +0000 UTC m=+2.972197426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:43.253551 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.253521    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkt5\" (UniqueName: \"kubernetes.io/projected/0053d784-2bcf-4674-a6fa-2301aa65cab3-kube-api-access-dfkt5\") pod \"node-ca-l4qff\" (UID: \"0053d784-2bcf-4674-a6fa-2301aa65cab3\") " pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.253660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.253641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvsgh\" (UniqueName: \"kubernetes.io/projected/672e4e80-f74f-4e3c-a20f-3167fe4970b5-kube-api-access-qvsgh\") pod \"node-resolver-lwjjd\" (UID: \"672e4e80-f74f-4e3c-a20f-3167fe4970b5\") " pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.253987 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.253921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgxr\" (UniqueName: \"kubernetes.io/projected/8948c7b9-90c8-40fe-ab4f-64f9a3e73776-kube-api-access-8fgxr\") pod \"multus-additional-cni-plugins-ngptk\" (UID: \"8948c7b9-90c8-40fe-ab4f-64f9a3e73776\") " pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.255956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.254329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhqd\" (UniqueName: \"kubernetes.io/projected/150f7de7-ecae-42c2-8942-dde8cc5c3203-kube-api-access-4lhqd\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.257411 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.257387    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrrp\" (UniqueName: \"kubernetes.io/projected/2d66e606-fc0c-4031-8314-7320e1ca9c74-kube-api-access-qzrrp\") pod \"ovnkube-node-7hx9v\" (UID: \"2d66e606-fc0c-4031-8314-7320e1ca9c74\") " pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.257499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.257477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz8d8\" (UniqueName: \"kubernetes.io/projected/b847b1ff-f1ba-47ce-80c4-5ff306240e68-kube-api-access-qz8d8\") pod \"aws-ebs-csi-driver-node-2pmcw\" (UID: \"b847b1ff-f1ba-47ce-80c4-5ff306240e68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.257599 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.257579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jc4\" (UniqueName: \"kubernetes.io/projected/76f64980-04b2-4e0b-8615-12c21a3f335b-kube-api-access-x8jc4\") pod \"multus-wfz4v\" (UID: \"76f64980-04b2-4e0b-8615-12c21a3f335b\") " pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.258913 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.258893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f5w\" (UniqueName: \"kubernetes.io/projected/27028e10-d16d-4940-8fb7-f76768ac35df-kube-api-access-h9f5w\") pod \"tuned-dfdtk\" (UID: \"27028e10-d16d-4940-8fb7-f76768ac35df\") " pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.313295 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.313263    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:09:43.339859 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.339817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal" event={"ID":"d4b510b9e2c520fe00d8e8ab6156268a","Type":"ContainerStarted","Data":"fd3c2c02fb0a23940329582ad3d13a4f54c152927f2914a308e1d2527648ceff"}
Jul 24 13:09:43.340034 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.339997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28f7t\" (UniqueName: \"kubernetes.io/projected/31bd44cc-1c81-46bd-9083-717dae54b5db-kube-api-access-28f7t\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.340096 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.340034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31bd44cc-1c81-46bd-9083-717dae54b5db-iptables-alerter-script\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.340350 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.340330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31bd44cc-1c81-46bd-9083-717dae54b5db-host-slash\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.340441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.340423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31bd44cc-1c81-46bd-9083-717dae54b5db-host-slash\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.340571 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.340553    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31bd44cc-1c81-46bd-9083-717dae54b5db-iptables-alerter-script\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.348777 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.348708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28f7t\" (UniqueName: \"kubernetes.io/projected/31bd44cc-1c81-46bd-9083-717dae54b5db-kube-api-access-28f7t\") pod \"iptables-alerter-wxt6p\" (UID: \"31bd44cc-1c81-46bd-9083-717dae54b5db\") " pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.420656 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.420624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw"
Jul 24 13:09:43.434656 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.434622    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wfz4v"
Jul 24 13:09:43.444913 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.444891    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lwjjd"
Jul 24 13:09:43.450097 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.450074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l4qff"
Jul 24 13:09:43.464036 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.464007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:09:43.472118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.472097    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfdtk"
Jul 24 13:09:43.479798 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.479777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ngptk"
Jul 24 13:09:43.487537 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.487518    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:09:43.493524 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.493505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxt6p"
Jul 24 13:09:43.743212 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.743126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:43.743212 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.743180    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:43.743435 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.743274    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:43.743435 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.743316    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:43.743435 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.743329    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:09:44.743313986 +0000 UTC m=+3.967797291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:43.743435 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.743365    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:09:44.743350589 +0000 UTC m=+3.967833897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:43.843862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:43.843829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:43.844068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.843988    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:43.844068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.844008    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:43.844068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.844021    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:43.844269 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:43.844082    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:09:44.844062548 +0000 UTC m=+4.068545853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:44.162265 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.162204    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:04:43 +0000 UTC" deadline="2028-02-10 22:58:13.908066612 +0000 UTC"
Jul 24 13:09:44.162265 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.162251    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13593h48m29.745820696s"
Jul 24 13:09:44.275710 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.275679    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0445648_3793_4fc2_80b6_1f0b2733536c.slice/crio-7e7b66d51502651345c2550c25fefc51c9f674ce04572ac88576b41b829a8210 WatchSource:0}: Error finding container 7e7b66d51502651345c2550c25fefc51c9f674ce04572ac88576b41b829a8210: Status 404 returned error can't find the container with id 7e7b66d51502651345c2550c25fefc51c9f674ce04572ac88576b41b829a8210
Jul 24 13:09:44.277110 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.277082    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27028e10_d16d_4940_8fb7_f76768ac35df.slice/crio-84791972887a934e7aaec8e336c3159a18095e3acb9238a11f291724af7bd685 WatchSource:0}: Error finding container 84791972887a934e7aaec8e336c3159a18095e3acb9238a11f291724af7bd685: Status 404 returned error can't find the container with id 84791972887a934e7aaec8e336c3159a18095e3acb9238a11f291724af7bd685
Jul 24 13:09:44.278824 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.278801    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb847b1ff_f1ba_47ce_80c4_5ff306240e68.slice/crio-6e8a1df51530b270e04c67cdc49f4d420ee2775402100822ae10dda5afbdac1c WatchSource:0}: Error finding container 6e8a1df51530b270e04c67cdc49f4d420ee2775402100822ae10dda5afbdac1c: Status 404 returned error can't find the container with id 6e8a1df51530b270e04c67cdc49f4d420ee2775402100822ae10dda5afbdac1c
Jul 24 13:09:44.282702 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.282678    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672e4e80_f74f_4e3c_a20f_3167fe4970b5.slice/crio-d0cd2e111249521111c6b6ad0abff3fd587e25937072b3a44aa4f546f0e0404e WatchSource:0}: Error finding container d0cd2e111249521111c6b6ad0abff3fd587e25937072b3a44aa4f546f0e0404e: Status 404 returned error can't find the container with id d0cd2e111249521111c6b6ad0abff3fd587e25937072b3a44aa4f546f0e0404e
Jul 24 13:09:44.287353 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.287114    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d66e606_fc0c_4031_8314_7320e1ca9c74.slice/crio-ecdc073da90296bd2e7f77fd36fa01c5b42f3f3356ae7cd417921c614ac58512 WatchSource:0}: Error finding container ecdc073da90296bd2e7f77fd36fa01c5b42f3f3356ae7cd417921c614ac58512: Status 404 returned error can't find the container with id ecdc073da90296bd2e7f77fd36fa01c5b42f3f3356ae7cd417921c614ac58512
Jul 24 13:09:44.287353 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:09:44.287330    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8948c7b9_90c8_40fe_ab4f_64f9a3e73776.slice/crio-66b261e400392d7f5311c6a76095aec7813ae5b931fdac2dd36c728d3c3d0f56 WatchSource:0}: Error finding container 66b261e400392d7f5311c6a76095aec7813ae5b931fdac2dd36c728d3c3d0f56: Status 404 returned error can't find the container with id 66b261e400392d7f5311c6a76095aec7813ae5b931fdac2dd36c728d3c3d0f56
Jul 24 13:09:44.334820 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.334795    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:44.334911 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.334886    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:44.342837 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.342704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw" event={"ID":"b847b1ff-f1ba-47ce-80c4-5ff306240e68","Type":"ContainerStarted","Data":"6e8a1df51530b270e04c67cdc49f4d420ee2775402100822ae10dda5afbdac1c"}
Jul 24 13:09:44.343909 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.343862    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerStarted","Data":"66b261e400392d7f5311c6a76095aec7813ae5b931fdac2dd36c728d3c3d0f56"}
Jul 24 13:09:44.344958 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.344871    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wfz4v" event={"ID":"76f64980-04b2-4e0b-8615-12c21a3f335b","Type":"ContainerStarted","Data":"c5640ef7c86629215697b0042d967790aad00af7db10879b267beb45b9e82c08"}
Jul 24 13:09:44.346138 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.346106    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l4qff" event={"ID":"0053d784-2bcf-4674-a6fa-2301aa65cab3","Type":"ContainerStarted","Data":"6743ffc81db5df69ed6e71597d3a121ed5c1dda020b842b95263952b62a7ab51"}
Jul 24 13:09:44.347055 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.347032    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfdtk" event={"ID":"27028e10-d16d-4940-8fb7-f76768ac35df","Type":"ContainerStarted","Data":"84791972887a934e7aaec8e336c3159a18095e3acb9238a11f291724af7bd685"}
Jul 24 13:09:44.348597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.348575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4z72" event={"ID":"f0445648-3793-4fc2-80b6-1f0b2733536c","Type":"ContainerStarted","Data":"7e7b66d51502651345c2550c25fefc51c9f674ce04572ac88576b41b829a8210"}
Jul 24 13:09:44.352587 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.350863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxt6p" event={"ID":"31bd44cc-1c81-46bd-9083-717dae54b5db","Type":"ContainerStarted","Data":"1218719521f79e8ddc2f54a415848e7a7fc8a8b0360714d8237401636f1f1559"}
Jul 24 13:09:44.352743 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.352717    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"ecdc073da90296bd2e7f77fd36fa01c5b42f3f3356ae7cd417921c614ac58512"}
Jul 24 13:09:44.354444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.354408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lwjjd" event={"ID":"672e4e80-f74f-4e3c-a20f-3167fe4970b5","Type":"ContainerStarted","Data":"d0cd2e111249521111c6b6ad0abff3fd587e25937072b3a44aa4f546f0e0404e"}
Jul 24 13:09:44.752379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.752267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:44.752379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.752353    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:44.752587 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.752443    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:44.752587 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.752501    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:44.752587 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.752517    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:09:46.752497039 +0000 UTC m=+5.976980347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:44.752587 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.752553    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:09:46.752536093 +0000 UTC m=+5.977019412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:44.853051 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:44.853008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:44.853209 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.853188    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:44.853209 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.853208    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:44.853333 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.853221    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:44.853333 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:44.853280    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:09:46.853262251 +0000 UTC m=+6.077745556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:45.162967 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.162839    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:04:43 +0000 UTC" deadline="2028-02-11 18:31:54.43121158 +0000 UTC"
Jul 24 13:09:45.162967 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.162901    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13613h22m9.268314757s"
Jul 24 13:09:45.337694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.337665    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:45.337864 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:45.337793    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:45.338269 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.338242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:45.338622 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:45.338391    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:45.372758 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.372723    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal" event={"ID":"fd517d74ad9c69ffc34f61061c12dc4c","Type":"ContainerStarted","Data":"b8d3aa3dcea083d9c19695b4c43191eca50f1b3efcd0826e56e031d06a631566"}
Jul 24 13:09:45.376558 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.376526    2556 generic.go:356] "Generic (PLEG): container finished" podID="d4b510b9e2c520fe00d8e8ab6156268a" containerID="b8c256a65caac530dbb0977660c6690e3c4236487e4cbe1b20313bcd2b784641" exitCode=0
Jul 24 13:09:45.376666 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.376578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal" event={"ID":"d4b510b9e2c520fe00d8e8ab6156268a","Type":"ContainerDied","Data":"b8c256a65caac530dbb0977660c6690e3c4236487e4cbe1b20313bcd2b784641"}
Jul 24 13:09:45.389952 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:45.389879    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-14.ec2.internal" podStartSLOduration=3.389861897 podStartE2EDuration="3.389861897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:09:45.389258756 +0000 UTC m=+4.613742087" watchObservedRunningTime="2026-07-24 13:09:45.389861897 +0000 UTC m=+4.614345224"
Jul 24 13:09:46.335654 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:46.335619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:46.336124 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.335753    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:46.389954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:46.389904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal" event={"ID":"d4b510b9e2c520fe00d8e8ab6156268a","Type":"ContainerStarted","Data":"57e24b2ce6c0a390b51bcad1d6171d7e8bd57b38a64e8863508d2f1786ddfc2c"}
Jul 24 13:09:46.769792 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:46.769620    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:46.769792 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:46.769679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:46.770057 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.769846    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:46.770057 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.769908    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:09:50.769889755 +0000 UTC m=+9.994373065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:46.770393 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.770321    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:46.770393 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.770375    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:09:50.770358571 +0000 UTC m=+9.994841878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:46.870818 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:46.870495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:46.870818 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.870669    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:46.870818 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.870689    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:46.870818 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.870702    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:46.870818 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:46.870767    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:09:50.870745325 +0000 UTC m=+10.095228633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:47.337716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:47.337684    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:47.338181 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:47.337804    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:47.338244 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:47.338199    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:47.338393 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:47.338302    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:48.335662 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:48.335631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:48.335844 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:48.335739    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:49.335895 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:49.335337    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:49.335895 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:49.335472    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:49.336449 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:49.336429    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:49.336561 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:49.336541    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:50.335147 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:50.335110    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:50.335345 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.335324    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:50.805272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:50.805180    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:50.805272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:50.805253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:50.805772 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.805415    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:50.805772 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.805478    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:09:58.805458578 +0000 UTC m=+18.029941904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:50.805887 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.805837    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:50.805887 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.805879    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:09:58.805863672 +0000 UTC m=+18.030346977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:50.907138 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:50.906539    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:50.907138 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.906709    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:50.907138 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.906730    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:50.907138 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.906742    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:50.907138 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:50.906803    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:09:58.906782647 +0000 UTC m=+18.131265975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:51.338095 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:51.338013    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:51.338326 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:51.338151    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:51.338390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:51.338333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:51.338843 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:51.338453    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:52.335146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:52.335116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:52.335618 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:52.335218    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:53.335500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:53.335033    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:53.335500 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:53.335167    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:53.336005 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:53.335601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:53.336005 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:53.335679    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:54.335069 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:54.335034    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:54.335228 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:54.335149    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:55.335498 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:55.335463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:55.335982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:55.335474    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:55.335982 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:55.335600    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:55.335982 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:55.335694    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:56.335268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:56.335230    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:56.335466 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:56.335358    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:57.335264 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:57.335231    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:57.335665 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:57.335285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:57.335665 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:57.335364    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:09:57.335665 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:57.335447    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:58.334998 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:58.334968    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:58.335260 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.335073    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:09:58.870630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:58.870600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:58.871068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:58.870650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:09:58.871068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.870738    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:58.871068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.870808    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:58.871068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.870811    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:10:14.870789137 +0000 UTC m=+34.095272455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:09:58.871068 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.870889    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:10:14.870866061 +0000 UTC m=+34.095349366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:09:58.971300 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:58.971267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:58.971469 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.971399    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:09:58.971469 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.971422    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:09:58.971469 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.971433    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:58.971581 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:58.971493    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:10:14.971472745 +0000 UTC m=+34.195956053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:09:59.335380 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:59.335300    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:09:59.335553 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:59.335423    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:09:59.335553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:09:59.335478    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:09:59.335656 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:09:59.335594    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:00.335695 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:00.335659    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:00.336145 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:00.335767    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:01.336377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:01.336328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:01.336776 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:01.336452    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:01.336776 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:01.336502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:01.336776 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:01.336623    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:02.334733 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:02.334698    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:02.334917 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:02.334828    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:03.337031 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.336900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:03.337604 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:03.337049    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:03.337604 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.337101    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:03.337604 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:03.337254    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:03.424106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.424044    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"96d42134c5f4f2fa0eda39f076373cdd415ddf76302de87960f1ac364b28e803"}
Jul 24 13:10:03.424205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.424162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"7cffe15896a4c194c17b2c7f36500fe894e31663dadc242621e078042dfc6e22"}
Jul 24 13:10:03.425564 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.425532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lwjjd" event={"ID":"672e4e80-f74f-4e3c-a20f-3167fe4970b5","Type":"ContainerStarted","Data":"4693abc920d5e53f27adfffc7ed64d19be4ebaee063f97229e950ecc26b77e58"}
Jul 24 13:10:03.427068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.427049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw" event={"ID":"b847b1ff-f1ba-47ce-80c4-5ff306240e68","Type":"ContainerStarted","Data":"43046aa2017819bf415f3a14e558086c65df821a2f24419f10bee3c95100034c"}
Jul 24 13:10:03.428403 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.428383    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="9351ec42b2d13c1ea5238fb8a6ec06e3d84081ca1dd334c7b135616d8647a311" exitCode=0
Jul 24 13:10:03.428500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.428470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"9351ec42b2d13c1ea5238fb8a6ec06e3d84081ca1dd334c7b135616d8647a311"}
Jul 24 13:10:03.432908 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.432883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wfz4v" event={"ID":"76f64980-04b2-4e0b-8615-12c21a3f335b","Type":"ContainerStarted","Data":"456f26ed1e547f8467aa066122765c5358c1d1a12ab42287a2c3d44fb2617187"}
Jul 24 13:10:03.434883 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.434863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l4qff" event={"ID":"0053d784-2bcf-4674-a6fa-2301aa65cab3","Type":"ContainerStarted","Data":"e216ac49cfa03c0bd61dc613da0a1cfda2b9d1bc2e4757836be9546d797e53d6"}
Jul 24 13:10:03.436992 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.436968    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfdtk" event={"ID":"27028e10-d16d-4940-8fb7-f76768ac35df","Type":"ContainerStarted","Data":"a831c7fad1d86631943818ecf0c1f82b761fba3f2d1464fc35d02811506227bc"}
Jul 24 13:10:03.438705 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.438670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4z72" event={"ID":"f0445648-3793-4fc2-80b6-1f0b2733536c","Type":"ContainerStarted","Data":"9d63c5bc8aab1ff6174a546963bfac7aa6b0bd5b38d403c397ab22cc1ffa9ab5"}
Jul 24 13:10:03.441362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.441309    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-14.ec2.internal" podStartSLOduration=21.441293422 podStartE2EDuration="21.441293422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:09:46.406053703 +0000 UTC m=+5.630537031" watchObservedRunningTime="2026-07-24 13:10:03.441293422 +0000 UTC m=+22.665776762"
Jul 24 13:10:03.441681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.441644    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lwjjd" podStartSLOduration=3.684468781 podStartE2EDuration="22.44163239s" totalImagesPullingTime="18.757163609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:03.440646047 +0000 UTC m=+22.665129376" watchObservedRunningTime="2026-07-24 13:10:03.44163239 +0000 UTC m=+22.666115738"
Jul 24 13:10:03.471069 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.471020    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dfdtk" podStartSLOduration=3.709450199 podStartE2EDuration="22.471005783s" totalImagesPullingTime="18.761555584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:03.470502551 +0000 UTC m=+22.694985877" watchObservedRunningTime="2026-07-24 13:10:03.471005783 +0000 UTC m=+22.695489110"
Jul 24 13:10:03.471317 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.471297    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l4qff" podStartSLOduration=8.324782485 podStartE2EDuration="22.471288559s" totalImagesPullingTime="14.146506074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:03.454736514 +0000 UTC m=+22.679219841" watchObservedRunningTime="2026-07-24 13:10:03.471288559 +0000 UTC m=+22.695771922"
Jul 24 13:10:03.504972 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.504914    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k4z72" podStartSLOduration=8.346604551 podStartE2EDuration="22.504899355s" totalImagesPullingTime="14.158294804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:03.504784192 +0000 UTC m=+22.729267519" watchObservedRunningTime="2026-07-24 13:10:03.504899355 +0000 UTC m=+22.729382682"
Jul 24 13:10:03.520982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:03.520924    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wfz4v" podStartSLOduration=3.737155388 podStartE2EDuration="22.520909527s" totalImagesPullingTime="18.783754139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:03.520630388 +0000 UTC m=+22.745113730" watchObservedRunningTime="2026-07-24 13:10:03.520909527 +0000 UTC m=+22.745392854"
Jul 24 13:10:04.335736 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.335505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:04.335946 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:04.335778    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:04.445445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.445420    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.445716    2556 generic.go:356] "Generic (PLEG): container finished" podID="2d66e606-fc0c-4031-8314-7320e1ca9c74" containerID="96d42134c5f4f2fa0eda39f076373cdd415ddf76302de87960f1ac364b28e803" exitCode=1
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.445810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerDied","Data":"96d42134c5f4f2fa0eda39f076373cdd415ddf76302de87960f1ac364b28e803"}
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.445905    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"32422b98b5bb2b61d57e58ed451501b8fecce7a328f0d47975d8b672d5385158"}
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.445975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"f4d2b44f14a306a74463a121141069a65acaf144c1a3024c1ef776d86a8bc2c4"}
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.446016    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"3dbe705419a640c649b307e3d46e2f119dbeab925ce1e8330fc7379241c895e4"}
Jul 24 13:10:04.446080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.446053    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"c002537e7c13d32f3ba206e34b3738cb8f9da5dc1c5113fff3c59587768ae585"}
Jul 24 13:10:04.887562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:04.887536    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 13:10:05.202317 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.202177    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T13:10:04.88755475Z","UUID":"303ceed1-e76e-4d6f-ba7d-7d075f1255e8","Handler":null,"Name":"","Endpoint":""}
Jul 24 13:10:05.203621 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.203603    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 13:10:05.203723 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.203638    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 13:10:05.335261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.335233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:05.335432 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:05.335343    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:05.335432 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.335391    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:05.335546 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:05.335474    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:05.450166 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.450093    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxt6p" event={"ID":"31bd44cc-1c81-46bd-9083-717dae54b5db","Type":"ContainerStarted","Data":"f77afafb69161f209c33510f97be7bf9d408800aca9775691c48ca86eabb4539"}
Jul 24 13:10:05.452212 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.452146    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw" event={"ID":"b847b1ff-f1ba-47ce-80c4-5ff306240e68","Type":"ContainerStarted","Data":"480fd64202f60c618b3b3683d4fda130aeb067a5d6f4905b8d5bd0c9e301f1ba"}
Jul 24 13:10:05.466744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:05.466667    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxt6p" podStartSLOduration=5.7371267889999995 podStartE2EDuration="24.46665455s" totalImagesPullingTime="18.729527761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:05.466032552 +0000 UTC m=+24.690515875" watchObservedRunningTime="2026-07-24 13:10:05.46665455 +0000 UTC m=+24.691137876"
Jul 24 13:10:06.036152 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.036123    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:10:06.037133 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.037101    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:10:06.335152 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.335080    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:06.335329 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:06.335200    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:06.459017 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.458895    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:10:06.459428 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.459338    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"110166942ddd1d6647ddd40b36a2c9bbd20f10c459b4caf08f42ace85b35750a"}
Jul 24 13:10:06.464505 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.464470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw" event={"ID":"b847b1ff-f1ba-47ce-80c4-5ff306240e68","Type":"ContainerStarted","Data":"7a04b659613bb609d35c67e096c9bc89889bd70b617b66c59436d3a663a7d6c2"}
Jul 24 13:10:06.487943 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:06.487888    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pmcw" podStartSLOduration=3.999294404 podStartE2EDuration="25.487875377s" totalImagesPullingTime="21.488580973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:06.487692892 +0000 UTC m=+25.712176222" watchObservedRunningTime="2026-07-24 13:10:06.487875377 +0000 UTC m=+25.712358704"
Jul 24 13:10:07.335774 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:07.335738    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:07.335984 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:07.335868    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:07.335984 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:07.335948    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:07.336109 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:07.336070    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:08.335830 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.335662    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:08.336482 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:08.335917    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:08.472663 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.472406    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:10:08.472816 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.472779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"b2998e0c9034dece99a9b942b3e25b7a12b870ac3b596dfaa0f36b9f05e4b426"}
Jul 24 13:10:08.473146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.473044    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:08.473279 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.473264    2556 scope.go:117] "RemoveContainer" containerID="96d42134c5f4f2fa0eda39f076373cdd415ddf76302de87960f1ac364b28e803"
Jul 24 13:10:08.474989 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.474958    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="80776a652c46e19dda779595be966e5e2971c78c2ad87e4f813bfece818f2983" exitCode=0
Jul 24 13:10:08.475100 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.475012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"80776a652c46e19dda779595be966e5e2971c78c2ad87e4f813bfece818f2983"}
Jul 24 13:10:08.490121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:08.490103    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:09.334960 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.334857    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:09.335109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.335052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:09.335252 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:09.335044    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:09.335410 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:09.335247    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:09.481057 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.481033    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:10:09.481408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.481341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" event={"ID":"2d66e606-fc0c-4031-8314-7320e1ca9c74","Type":"ContainerStarted","Data":"151e0345cbf5478e49e2a4ae7cf5555a181302243a88020c417e84e8f0e9561b"}
Jul 24 13:10:09.481513 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.481499    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:09.481566 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.481555    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:09.496475 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.496454    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:09.507215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:09.507180    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v" podStartSLOduration=9.685889915 podStartE2EDuration="28.507165596s" totalImagesPullingTime="18.821275681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:09.50631358 +0000 UTC m=+28.730796907" watchObservedRunningTime="2026-07-24 13:10:09.507165596 +0000 UTC m=+28.731648917"
Jul 24 13:10:10.126139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.126028    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g75k9"]
Jul 24 13:10:10.126139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.126129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:10.126299 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:10.126230    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:10.127892 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.127662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsqdv"]
Jul 24 13:10:10.127892 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.127768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:10.127892 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:10.127876    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:10.128284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.128263    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mp7r"]
Jul 24 13:10:10.128376 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.128364    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:10.128460 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:10.128439    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:10.486082 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.485996    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="0cf43f155993fdab4272f47bcb618f3b7024e77e1d20046ad793d75d1343c4e0" exitCode=0
Jul 24 13:10:10.486450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:10.486079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"0cf43f155993fdab4272f47bcb618f3b7024e77e1d20046ad793d75d1343c4e0"}
Jul 24 13:10:11.336426 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:11.336260    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:11.336568 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:11.336477    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:12.335063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.335037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:12.335463 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.335079    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:12.335463 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:12.335160    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:12.335463 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:12.335256    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:12.492422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.492342    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="1e816db687aff528996f6df8cddbec501d2a8e9e49cc1e689a587a879f437414" exitCode=0
Jul 24 13:10:12.492422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.492404    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"1e816db687aff528996f6df8cddbec501d2a8e9e49cc1e689a587a879f437414"}
Jul 24 13:10:12.985408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.985376    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:10:12.985553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.985519    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 13:10:12.986012 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:12.985992    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4z72"
Jul 24 13:10:13.335728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:13.335694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:13.336378 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:13.335833    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:14.335155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:14.335122    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:14.335319 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:14.335135    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:14.335319 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.335247    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5mp7r" podUID="2ec6639f-9950-4bd0-85d5-a3542940b6aa"
Jul 24 13:10:14.335319 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.335305    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsqdv" podUID="150f7de7-ecae-42c2-8942-dde8cc5c3203"
Jul 24 13:10:14.892356 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:14.892324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:14.892890 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:14.892365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:14.892890 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.892476    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:10:14.892890 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.892486    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:10:14.892890 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.892541    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs podName:150f7de7-ecae-42c2-8942-dde8cc5c3203 nodeName:}" failed. No retries permitted until 2026-07-24 13:10:46.892522391 +0000 UTC m=+66.117005695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs") pod "network-metrics-daemon-rsqdv" (UID: "150f7de7-ecae-42c2-8942-dde8cc5c3203") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:10:14.892890 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.892555    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret podName:2ec6639f-9950-4bd0-85d5-a3542940b6aa nodeName:}" failed. No retries permitted until 2026-07-24 13:10:46.892548819 +0000 UTC m=+66.117032123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret") pod "global-pull-secret-syncer-5mp7r" (UID: "2ec6639f-9950-4bd0-85d5-a3542940b6aa") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:10:14.993391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:14.993362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:14.993554 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.993508    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:10:14.993554 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.993527    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:10:14.993554 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.993540    2556 projected.go:196] Error preparing data for projected volume kube-api-access-pnggf for pod openshift-network-diagnostics/network-check-target-g75k9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:10:14.993654 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:14.993592    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf podName:1d80860e-1272-4cc4-8997-69b40184e1ca nodeName:}" failed. No retries permitted until 2026-07-24 13:10:46.993579037 +0000 UTC m=+66.218062342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pnggf" (UniqueName: "kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf") pod "network-check-target-g75k9" (UID: "1d80860e-1272-4cc4-8997-69b40184e1ca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:10:15.335502 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:15.335421    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:15.335656 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:15.335536    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g75k9" podUID="1d80860e-1272-4cc4-8997-69b40184e1ca"
Jul 24 13:10:16.091084 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.091010    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-14.ec2.internal" event="NodeReady"
Jul 24 13:10:16.091626 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.091171    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 13:10:16.138524 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.138488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vrjqx"]
Jul 24 13:10:16.170442 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.170395    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lkbk9"]
Jul 24 13:10:16.170601 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.170533    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.172998 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.172979    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 13:10:16.173146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.173020    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 13:10:16.173502 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.173074    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 13:10:16.173502 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.173127    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xhz7z\""
Jul 24 13:10:16.173502 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.173134    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 13:10:16.184600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.184583    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkbk9"]
Jul 24 13:10:16.184694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.184616    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrjqx"]
Jul 24 13:10:16.184694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.184643    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rvjrb"]
Jul 24 13:10:16.184694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.184687    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.187872 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.187852    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 13:10:16.187872 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.187869    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n54jn\""
Jul 24 13:10:16.188093 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.188046    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 13:10:16.202723 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.202702    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvjrb"]
Jul 24 13:10:16.202821 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.202793    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.204806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.204786    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 13:10:16.204806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.204798    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t8pk6\""
Jul 24 13:10:16.205050 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.204879    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 13:10:16.205050 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.205047    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 13:10:16.302680 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gqm\" (UniqueName: \"kubernetes.io/projected/be8f59de-48d2-49f1-b36e-846c6ccfab47-kube-api-access-24gqm\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.302877 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f46777f9-0473-4e96-9624-bdd5fb137698-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.302877 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8f59de-48d2-49f1-b36e-846c6ccfab47-metrics-tls\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.302877 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f46777f9-0473-4e96-9624-bdd5fb137698-data-volume\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.302877 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302760    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f46777f9-0473-4e96-9624-bdd5fb137698-crio-socket\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.302877 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-cert\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.303125 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66cmj\" (UniqueName: \"kubernetes.io/projected/f46777f9-0473-4e96-9624-bdd5fb137698-kube-api-access-66cmj\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.303125 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be8f59de-48d2-49f1-b36e-846c6ccfab47-config-volume\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.303125 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.302992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8f59de-48d2-49f1-b36e-846c6ccfab47-tmp-dir\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.303125 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.303067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f46777f9-0473-4e96-9624-bdd5fb137698-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.303125 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.303094    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6drjx\" (UniqueName: \"kubernetes.io/projected/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-kube-api-access-6drjx\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.335312 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.335276    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:16.335473 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.335276    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:16.337412 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.337390    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:10:16.337539 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.337457    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:10:16.337539 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.337460    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rh25q\""
Jul 24 13:10:16.403803 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be8f59de-48d2-49f1-b36e-846c6ccfab47-config-volume\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.403803 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8f59de-48d2-49f1-b36e-846c6ccfab47-tmp-dir\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f46777f9-0473-4e96-9624-bdd5fb137698-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6drjx\" (UniqueName: \"kubernetes.io/projected/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-kube-api-access-6drjx\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gqm\" (UniqueName: \"kubernetes.io/projected/be8f59de-48d2-49f1-b36e-846c6ccfab47-kube-api-access-24gqm\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f46777f9-0473-4e96-9624-bdd5fb137698-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.403996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8f59de-48d2-49f1-b36e-846c6ccfab47-metrics-tls\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404022    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f46777f9-0473-4e96-9624-bdd5fb137698-data-volume\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.404064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f46777f9-0473-4e96-9624-bdd5fb137698-crio-socket\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.404395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-cert\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.404395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66cmj\" (UniqueName: \"kubernetes.io/projected/f46777f9-0473-4e96-9624-bdd5fb137698-kube-api-access-66cmj\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.404395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404242    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8f59de-48d2-49f1-b36e-846c6ccfab47-tmp-dir\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.404485 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be8f59de-48d2-49f1-b36e-846c6ccfab47-config-volume\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.404793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.404768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f46777f9-0473-4e96-9624-bdd5fb137698-crio-socket\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.405117 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.405079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f46777f9-0473-4e96-9624-bdd5fb137698-data-volume\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.405216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.405200    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f46777f9-0473-4e96-9624-bdd5fb137698-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.410954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.408970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8f59de-48d2-49f1-b36e-846c6ccfab47-metrics-tls\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.410954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.409158    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f46777f9-0473-4e96-9624-bdd5fb137698-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.410954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.409903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-cert\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.412724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.412703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6drjx\" (UniqueName: \"kubernetes.io/projected/1c6ed8d2-0deb-404f-ab68-6972a2ec9d03-kube-api-access-6drjx\") pod \"ingress-canary-rvjrb\" (UID: \"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03\") " pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.412834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.412762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66cmj\" (UniqueName: \"kubernetes.io/projected/f46777f9-0473-4e96-9624-bdd5fb137698-kube-api-access-66cmj\") pod \"insights-runtime-extractor-vrjqx\" (UID: \"f46777f9-0473-4e96-9624-bdd5fb137698\") " pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.412894 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.412859    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gqm\" (UniqueName: \"kubernetes.io/projected/be8f59de-48d2-49f1-b36e-846c6ccfab47-kube-api-access-24gqm\") pod \"dns-default-lkbk9\" (UID: \"be8f59de-48d2-49f1-b36e-846c6ccfab47\") " pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.480784 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.480746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vrjqx"
Jul 24 13:10:16.493920 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.493892    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:16.511443 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.511411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvjrb"
Jul 24 13:10:16.913147 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.912984    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkbk9"]
Jul 24 13:10:16.916530 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.916504    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvjrb"]
Jul 24 13:10:16.928666 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:16.928640    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vrjqx"]
Jul 24 13:10:16.928784 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:16.928664    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8f59de_48d2_49f1_b36e_846c6ccfab47.slice/crio-d515d3ff28ffb3aa7bdcabffcd94423aab26a8acab371f10927aa2a10d71d832 WatchSource:0}: Error finding container d515d3ff28ffb3aa7bdcabffcd94423aab26a8acab371f10927aa2a10d71d832: Status 404 returned error can't find the container with id d515d3ff28ffb3aa7bdcabffcd94423aab26a8acab371f10927aa2a10d71d832
Jul 24 13:10:16.929583 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:16.929511    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6ed8d2_0deb_404f_ab68_6972a2ec9d03.slice/crio-1d017cbb8c4f3a1ec0d9f9c0a43081b367aae3713c4e6cfae2f995af795cc52d WatchSource:0}: Error finding container 1d017cbb8c4f3a1ec0d9f9c0a43081b367aae3713c4e6cfae2f995af795cc52d: Status 404 returned error can't find the container with id 1d017cbb8c4f3a1ec0d9f9c0a43081b367aae3713c4e6cfae2f995af795cc52d
Jul 24 13:10:16.932377 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:16.932354    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46777f9_0473_4e96_9624_bdd5fb137698.slice/crio-517825fba0277c1a7951d0ddc2ec2be03f322e4f8dd8a72d06aefc8ad8c499dc WatchSource:0}: Error finding container 517825fba0277c1a7951d0ddc2ec2be03f322e4f8dd8a72d06aefc8ad8c499dc: Status 404 returned error can't find the container with id 517825fba0277c1a7951d0ddc2ec2be03f322e4f8dd8a72d06aefc8ad8c499dc
Jul 24 13:10:17.335871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.335790    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:17.338779 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.338751    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 13:10:17.338779 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.338773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dqwnd\""
Jul 24 13:10:17.340148 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.340126    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 13:10:17.505478 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.505438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkbk9" event={"ID":"be8f59de-48d2-49f1-b36e-846c6ccfab47","Type":"ContainerStarted","Data":"d515d3ff28ffb3aa7bdcabffcd94423aab26a8acab371f10927aa2a10d71d832"}
Jul 24 13:10:17.506728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.506690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvjrb" event={"ID":"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03","Type":"ContainerStarted","Data":"1d017cbb8c4f3a1ec0d9f9c0a43081b367aae3713c4e6cfae2f995af795cc52d"}
Jul 24 13:10:17.508027 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.508002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrjqx" event={"ID":"f46777f9-0473-4e96-9624-bdd5fb137698","Type":"ContainerStarted","Data":"517825fba0277c1a7951d0ddc2ec2be03f322e4f8dd8a72d06aefc8ad8c499dc"}
Jul 24 13:10:17.511062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.511040    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="77c5d79391b042a75ed59239b99b96d7f04a9e3cff85f73de7c0afa8c5117c54" exitCode=0
Jul 24 13:10:17.511173 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:17.511072    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"77c5d79391b042a75ed59239b99b96d7f04a9e3cff85f73de7c0afa8c5117c54"}
Jul 24 13:10:18.517467 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:18.517435    2556 generic.go:356] "Generic (PLEG): container finished" podID="8948c7b9-90c8-40fe-ab4f-64f9a3e73776" containerID="9c2a563e0e72b9fd112d7156c6d1cac717e31738e71d72985bdf0f27abc94ce4" exitCode=0
Jul 24 13:10:18.517885 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:18.517489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerDied","Data":"9c2a563e0e72b9fd112d7156c6d1cac717e31738e71d72985bdf0f27abc94ce4"}
Jul 24 13:10:20.528637 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:20.528412    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngptk" event={"ID":"8948c7b9-90c8-40fe-ab4f-64f9a3e73776","Type":"ContainerStarted","Data":"71158b7b0ad1c3fa07131a7c67b4b713e351bdad4082c6e6fe47c58dc8ee868e"}
Jul 24 13:10:20.530033 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:20.530000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkbk9" event={"ID":"be8f59de-48d2-49f1-b36e-846c6ccfab47","Type":"ContainerStarted","Data":"f7213b2620cfc5e3cf52af2834f1d5630f4d98ff86fa0a0cb86f53162910c2e4"}
Jul 24 13:10:20.555518 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:20.555447    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ngptk" podStartSLOduration=9.423081353 podStartE2EDuration="39.555426042s" totalImagesPullingTime="30.132344689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:20.55254309 +0000 UTC m=+39.777026421" watchObservedRunningTime="2026-07-24 13:10:20.555426042 +0000 UTC m=+39.779909371"
Jul 24 13:10:23.539682 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:23.539644    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkbk9" event={"ID":"be8f59de-48d2-49f1-b36e-846c6ccfab47","Type":"ContainerStarted","Data":"619e1ec190adbdf2bfd00521cd9bdcf32816b66295cce165b3d025ba57315322"}
Jul 24 13:10:23.540116 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:23.539730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:23.541190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:23.541164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvjrb" event={"ID":"1c6ed8d2-0deb-404f-ab68-6972a2ec9d03","Type":"ContainerStarted","Data":"4e8652df851439c79e6214c1eaa00801565b76f3ce74f7b3ac21583439c46909"}
Jul 24 13:10:23.556063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:23.556020    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lkbk9" podStartSLOduration=4.486162416 podStartE2EDuration="7.555992949s" totalImagesPullingTime="3.069830533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:23.554541378 +0000 UTC m=+42.779024704" watchObservedRunningTime="2026-07-24 13:10:23.555992949 +0000 UTC m=+42.780476278"
Jul 24 13:10:23.567838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:23.567800    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rvjrb" podStartSLOduration=4.492747058 podStartE2EDuration="7.567789214s" totalImagesPullingTime="3.075042156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:23.567736717 +0000 UTC m=+42.792220043" watchObservedRunningTime="2026-07-24 13:10:23.567789214 +0000 UTC m=+42.792272540"
Jul 24 13:10:24.546686 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:24.546648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrjqx" event={"ID":"f46777f9-0473-4e96-9624-bdd5fb137698","Type":"ContainerStarted","Data":"fa42a4cc0347937825a1b38d47c516c4fd4fb7e84e55252579be9fd4ff2f8e51"}
Jul 24 13:10:27.556615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:27.556582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrjqx" event={"ID":"f46777f9-0473-4e96-9624-bdd5fb137698","Type":"ContainerStarted","Data":"b0280d3035ad8e7bfe73137770d817cd82f0afaf75280e1ecc2a2419a5fb0575"}
Jul 24 13:10:30.549068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:30.549033    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lkbk9"
Jul 24 13:10:31.574149 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.574118    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vrjqx" event={"ID":"f46777f9-0473-4e96-9624-bdd5fb137698","Type":"ContainerStarted","Data":"0c8fecacc06b714feb4767d6de3c4392a3f3b40b7ee4dcc9b0b46cca005dd089"}
Jul 24 13:10:31.592860 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.592815    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vrjqx" podStartSLOduration=1.451149706 podStartE2EDuration="15.592800161s" totalImagesPullingTime="14.141650455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:10:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:31.591493365 +0000 UTC m=+50.815976691" watchObservedRunningTime="2026-07-24 13:10:31.592800161 +0000 UTC m=+50.817283488"
Jul 24 13:10:31.766291 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.766257    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"]
Jul 24 13:10:31.785097 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.785069    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"]
Jul 24 13:10:31.785241 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.785171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:31.787122 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.787095    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 13:10:31.787243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.787102    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t555l\""
Jul 24 13:10:31.814444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.814419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-866xv\" (UID: \"7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:31.915230 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.915150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-866xv\" (UID: \"7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:31.919278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:31.919255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-866xv\" (UID: \"7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:32.096374 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:32.096346    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:32.240405 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:32.240373    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"]
Jul 24 13:10:32.243096 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:32.243070    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6b3bd8_de21_4dd1_80d6_b8acc2a6ddce.slice/crio-0868a321942b072e573b1d5520e3a573d6bcfaac1806efa5a936d69920ffb5b3 WatchSource:0}: Error finding container 0868a321942b072e573b1d5520e3a573d6bcfaac1806efa5a936d69920ffb5b3: Status 404 returned error can't find the container with id 0868a321942b072e573b1d5520e3a573d6bcfaac1806efa5a936d69920ffb5b3
Jul 24 13:10:32.578413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:32.578374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv" event={"ID":"7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce","Type":"ContainerStarted","Data":"0868a321942b072e573b1d5520e3a573d6bcfaac1806efa5a936d69920ffb5b3"}
Jul 24 13:10:34.586004 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.585963    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv" event={"ID":"7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce","Type":"ContainerStarted","Data":"452a9da4e15ef884da04e3b04aaf5f28dbc80fc5a94c1c559cfecea49d9fc75b"}
Jul 24 13:10:34.586466 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.586180    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:34.590798 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.590777    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv"
Jul 24 13:10:34.600392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.600353    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-866xv" podStartSLOduration=1.884337731 podStartE2EDuration="3.60033968s" totalImagesPullingTime="1.716001949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:34.599547408 +0000 UTC m=+53.824030735" watchObservedRunningTime="2026-07-24 13:10:34.60033968 +0000 UTC m=+53.824823007"
Jul 24 13:10:34.831185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.831149    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"]
Jul 24 13:10:34.850807 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.850740    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"]
Jul 24 13:10:34.850982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.850853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:34.854215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854117    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 13:10:34.854215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854137    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 13:10:34.854215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 13:10:34.854215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854151    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 13:10:34.854494 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854128    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 13:10:34.854494 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.854444    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hjjjc\""
Jul 24 13:10:34.936213 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.936179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plggd\" (UniqueName: \"kubernetes.io/projected/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-kube-api-access-plggd\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:34.936364 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.936224    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:34.936364 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.936243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:34.936364 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:34.936322    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.037195 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.037161    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plggd\" (UniqueName: \"kubernetes.io/projected/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-kube-api-access-plggd\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.037348 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.037210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.037348 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.037232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.037348 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.037287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.037963 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.037915    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.039684 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.039666    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.039828 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.039811    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.045070 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.045046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plggd\" (UniqueName: \"kubernetes.io/projected/eb2564cb-7867-49fb-ac14-2f5a3aec6cff-kube-api-access-plggd\") pod \"prometheus-operator-84d69f794d-hm4mf\" (UID: \"eb2564cb-7867-49fb-ac14-2f5a3aec6cff\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.160228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.160152    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"
Jul 24 13:10:35.294849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.294818    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hm4mf"]
Jul 24 13:10:35.297697 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:35.297670    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2564cb_7867_49fb_ac14_2f5a3aec6cff.slice/crio-83e98c206cf2ecf8da08e48aa87c25efbdc3958f952a0d9ef3e0ae2a4dc6e920 WatchSource:0}: Error finding container 83e98c206cf2ecf8da08e48aa87c25efbdc3958f952a0d9ef3e0ae2a4dc6e920: Status 404 returned error can't find the container with id 83e98c206cf2ecf8da08e48aa87c25efbdc3958f952a0d9ef3e0ae2a4dc6e920
Jul 24 13:10:35.590162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:35.590130    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf" event={"ID":"eb2564cb-7867-49fb-ac14-2f5a3aec6cff","Type":"ContainerStarted","Data":"83e98c206cf2ecf8da08e48aa87c25efbdc3958f952a0d9ef3e0ae2a4dc6e920"}
Jul 24 13:10:37.600584 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:37.600554    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf" event={"ID":"eb2564cb-7867-49fb-ac14-2f5a3aec6cff","Type":"ContainerStarted","Data":"7b4abe054bd6707f89fa082a8b543fd2f584d216fab7573ca45de267d801b8e3"}
Jul 24 13:10:37.601010 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:37.600603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf" event={"ID":"eb2564cb-7867-49fb-ac14-2f5a3aec6cff","Type":"ContainerStarted","Data":"3dc7f54f10bf8cb7e0566c07de84fae026ff3f7b5722c579ffef8fdea01e58d0"}
Jul 24 13:10:37.618946 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:37.618882    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-hm4mf" podStartSLOduration=1.819018483 podStartE2EDuration="3.618866551s" totalImagesPullingTime="1.799848068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:37.618277983 +0000 UTC m=+56.842761311" watchObservedRunningTime="2026-07-24 13:10:37.618866551 +0000 UTC m=+56.843349879"
Jul 24 13:10:39.266385 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.266350    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7sq6g"]
Jul 24 13:10:39.289459 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.289435    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.291848 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.291825    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8tk55\""
Jul 24 13:10:39.292219 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.292205    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 13:10:39.292398 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.292386    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 13:10:39.292689 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.292667    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 13:10:39.369144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-sys\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-wtmp\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-tls\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpld\" (UniqueName: \"kubernetes.io/projected/35185b15-9219-46c2-88c3-cb3c2ad91761-kube-api-access-hrpld\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-metrics-client-ca\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-root\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369566 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.369566 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.369378    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-textfile\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469698 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469668    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-wtmp\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-tls\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469728    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpld\" (UniqueName: \"kubernetes.io/projected/35185b15-9219-46c2-88c3-cb3c2ad91761-kube-api-access-hrpld\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-metrics-client-ca\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-root\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.469871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-wtmp\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-root\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.469977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-textfile\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470127    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-sys\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35185b15-9219-46c2-88c3-cb3c2ad91761-sys\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-textfile\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.470441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.470431    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35185b15-9219-46c2-88c3-cb3c2ad91761-metrics-client-ca\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.472312 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.472292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.472409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.472316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35185b15-9219-46c2-88c3-cb3c2ad91761-node-exporter-tls\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.477272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.477243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpld\" (UniqueName: \"kubernetes.io/projected/35185b15-9219-46c2-88c3-cb3c2ad91761-kube-api-access-hrpld\") pod \"node-exporter-7sq6g\" (UID: \"35185b15-9219-46c2-88c3-cb3c2ad91761\") " pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.598809 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:39.598776    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sq6g"
Jul 24 13:10:39.639717 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:39.639686    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35185b15_9219_46c2_88c3_cb3c2ad91761.slice/crio-a7d8e4ff797d4b182de119a1c906d0ab665882064dc9bbb39ce9a69a5c259f7b WatchSource:0}: Error finding container a7d8e4ff797d4b182de119a1c906d0ab665882064dc9bbb39ce9a69a5c259f7b: Status 404 returned error can't find the container with id a7d8e4ff797d4b182de119a1c906d0ab665882064dc9bbb39ce9a69a5c259f7b
Jul 24 13:10:40.612691 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:40.612650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sq6g" event={"ID":"35185b15-9219-46c2-88c3-cb3c2ad91761","Type":"ContainerStarted","Data":"a7d8e4ff797d4b182de119a1c906d0ab665882064dc9bbb39ce9a69a5c259f7b"}
Jul 24 13:10:41.401664 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.401614    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"]
Jul 24 13:10:41.415516 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.415495    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.416646 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.416606    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"]
Jul 24 13:10:41.417769 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417748    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p4qcj\""
Jul 24 13:10:41.417908 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417888    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 13:10:41.417985 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 13:10:41.417985 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 13:10:41.417985 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417923    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 13:10:41.418135 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.417908    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-pgon4n1spbao\""
Jul 24 13:10:41.418135 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.418051    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 13:10:41.484421 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484388    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484421 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-grpc-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblz8\" (UniqueName: \"kubernetes.io/projected/3783e967-19bc-4305-857b-776834ea1d54-kube-api-access-kblz8\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484563    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484658 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484726 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.484726 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.484716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3783e967-19bc-4305-857b-776834ea1d54-metrics-client-ca\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.504474 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.504455    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7hx9v"
Jul 24 13:10:41.585515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585472    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3783e967-19bc-4305-857b-776834ea1d54-metrics-client-ca\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-grpc-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblz8\" (UniqueName: \"kubernetes.io/projected/3783e967-19bc-4305-857b-776834ea1d54-kube-api-access-kblz8\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.585804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.585778    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.586813 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.586790    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3783e967-19bc-4305-857b-776834ea1d54-metrics-client-ca\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.588586 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.588561    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.588692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.588630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.588809 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.588786    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.588847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.588824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.588982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.588964    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-thanos-querier-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.589023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.589011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3783e967-19bc-4305-857b-776834ea1d54-secret-grpc-tls\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.594417 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.594396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblz8\" (UniqueName: \"kubernetes.io/projected/3783e967-19bc-4305-857b-776834ea1d54-kube-api-access-kblz8\") pod \"thanos-querier-6fd9ff656c-f9tpf\" (UID: \"3783e967-19bc-4305-857b-776834ea1d54\") " pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.617212 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.617177    2556 generic.go:356] "Generic (PLEG): container finished" podID="35185b15-9219-46c2-88c3-cb3c2ad91761" containerID="46b31c3a001d16028945b252c072b487a75e2c90c5bd5971c99750308b1da642" exitCode=0
Jul 24 13:10:41.617545 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.617243    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sq6g" event={"ID":"35185b15-9219-46c2-88c3-cb3c2ad91761","Type":"ContainerDied","Data":"46b31c3a001d16028945b252c072b487a75e2c90c5bd5971c99750308b1da642"}
Jul 24 13:10:41.727186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.727148    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:41.875518 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:41.875416    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"]
Jul 24 13:10:41.878731 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:41.878704    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3783e967_19bc_4305_857b_776834ea1d54.slice/crio-bc5a716f762fb41eb933c54d0480e48e55573b29ef72ece5c28cc3ea60dea041 WatchSource:0}: Error finding container bc5a716f762fb41eb933c54d0480e48e55573b29ef72ece5c28cc3ea60dea041: Status 404 returned error can't find the container with id bc5a716f762fb41eb933c54d0480e48e55573b29ef72ece5c28cc3ea60dea041
Jul 24 13:10:42.622186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:42.622154    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"bc5a716f762fb41eb933c54d0480e48e55573b29ef72ece5c28cc3ea60dea041"}
Jul 24 13:10:42.627597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:42.627563    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sq6g" event={"ID":"35185b15-9219-46c2-88c3-cb3c2ad91761","Type":"ContainerStarted","Data":"b0556899b7dce895f2f0a3149f2f46891d73e3eab8a4171c16be835468511964"}
Jul 24 13:10:42.627728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:42.627619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sq6g" event={"ID":"35185b15-9219-46c2-88c3-cb3c2ad91761","Type":"ContainerStarted","Data":"115ad69c7dae2d6e7f8d99ae40c73943678e6e349d66f5aa1cac56c35f6dd5ca"}
Jul 24 13:10:42.646640 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:42.646590    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7sq6g" podStartSLOduration=2.102124922 podStartE2EDuration="3.646572168s" totalImagesPullingTime="1.436492175s" totalInitContainerRuntime="107.955071ms" isStatefulPod=true podCreationTimestamp="2026-07-24 13:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:42.64549053 +0000 UTC m=+61.869973858" watchObservedRunningTime="2026-07-24 13:10:42.646572168 +0000 UTC m=+61.871055496"
Jul 24 13:10:43.640633 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.640590    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"]
Jul 24 13:10:43.642836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.642816    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.644972 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.644948    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 13:10:43.645917 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.645894    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-buumldqd2hqgs\""
Jul 24 13:10:43.646042 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.646009    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 13:10:43.646105 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.646010    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 13:10:43.646105 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.646091    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 13:10:43.646210 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.646164    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g8kvj\""
Jul 24 13:10:43.655959 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.655231    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"]
Jul 24 13:10:43.704007 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.703972    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-client-certs\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704007 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-metrics-server-audit-profiles\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwgh\" (UniqueName: \"kubernetes.io/projected/8ccc7612-b377-4303-903f-93b61d3d3725-kube-api-access-hnwgh\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-tls\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-client-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704345    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.704415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.704372    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7612-b377-4303-903f-93b61d3d3725-audit-log\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805365 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7612-b377-4303-903f-93b61d3d3725-audit-log\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-client-certs\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805497 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805439    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-metrics-server-audit-profiles\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805497 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwgh\" (UniqueName: \"kubernetes.io/projected/8ccc7612-b377-4303-903f-93b61d3d3725-kube-api-access-hnwgh\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805594 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-tls\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805594 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-client-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.805867 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.805810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7612-b377-4303-903f-93b61d3d3725-audit-log\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.806118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.806091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.806495 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.806473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8ccc7612-b377-4303-903f-93b61d3d3725-metrics-server-audit-profiles\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.808431 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.808407    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-client-ca-bundle\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.808687 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.808668    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-tls\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.808759 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.808697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8ccc7612-b377-4303-903f-93b61d3d3725-secret-metrics-server-client-certs\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.814576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.814548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwgh\" (UniqueName: \"kubernetes.io/projected/8ccc7612-b377-4303-903f-93b61d3d3725-kube-api-access-hnwgh\") pod \"metrics-server-cdf6bbc7c-6qfrt\" (UID: \"8ccc7612-b377-4303-903f-93b61d3d3725\") " pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.954963 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.954891    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:10:43.999645 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:43.999541    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"]
Jul 24 13:10:44.002156 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.002130    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.004741 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.004517    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w7cmv\""
Jul 24 13:10:44.004865 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.004834    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 13:10:44.014648 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.014624    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"]
Jul 24 13:10:44.091596 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.091567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"]
Jul 24 13:10:44.095835 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:44.095809    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccc7612_b377_4303_903f_93b61d3d3725.slice/crio-a14e77c2682790536bc049d25d90dbf01906dda33d37f324893856ab5f724a8f WatchSource:0}: Error finding container a14e77c2682790536bc049d25d90dbf01906dda33d37f324893856ab5f724a8f: Status 404 returned error can't find the container with id a14e77c2682790536bc049d25d90dbf01906dda33d37f324893856ab5f724a8f
Jul 24 13:10:44.108460 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.108438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28sp\" (UniqueName: \"kubernetes.io/projected/731e5b78-6d8a-4388-9319-ccee65fc8bfc-kube-api-access-t28sp\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.108536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.108476    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.208917 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.208836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28sp\" (UniqueName: \"kubernetes.io/projected/731e5b78-6d8a-4388-9319-ccee65fc8bfc-kube-api-access-t28sp\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.208917 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.208886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.209114 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:44.209066    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 24 13:10:44.209159 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:10:44.209124    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert podName:731e5b78-6d8a-4388-9319-ccee65fc8bfc nodeName:}" failed. No retries permitted until 2026-07-24 13:10:44.709109328 +0000 UTC m=+63.933592633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-dtkv6" (UID: "731e5b78-6d8a-4388-9319-ccee65fc8bfc") : secret "monitoring-plugin-cert" not found
Jul 24 13:10:44.220895 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.220875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28sp\" (UniqueName: \"kubernetes.io/projected/731e5b78-6d8a-4388-9319-ccee65fc8bfc-kube-api-access-t28sp\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.476590 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.476510    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dff785474-x9xnb"]
Jul 24 13:10:44.479198 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.479164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.481470 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.481449    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 13:10:44.481585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.481553    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 13:10:44.481585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.481556    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 13:10:44.481585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.481556    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rbxx9\""
Jul 24 13:10:44.481978 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.481960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 13:10:44.482070 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.482054    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 13:10:44.487706 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.487689    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 13:10:44.500450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.500432    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dff785474-x9xnb"]
Jul 24 13:10:44.511588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511568    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-federate-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-metrics-client-ca\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511690    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfxg\" (UniqueName: \"kubernetes.io/projected/680d50cc-d6f5-43de-955e-c286d1086113-kube-api-access-scfxg\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-serving-certs-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.511906 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.511853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613005 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.612964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613115    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-federate-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-metrics-client-ca\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613181    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613400 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfxg\" (UniqueName: \"kubernetes.io/projected/680d50cc-d6f5-43de-955e-c286d1086113-kube-api-access-scfxg\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613400 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-serving-certs-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.613400 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.613297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.614317 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.614286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-metrics-client-ca\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.614442 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.614410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.614911 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.614864    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/680d50cc-d6f5-43de-955e-c286d1086113-serving-certs-ca-bundle\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.616237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.616179    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.616343 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.616312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-federate-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.616472 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.616345    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-secret-telemeter-client\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.616773 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.616756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/680d50cc-d6f5-43de-955e-c286d1086113-telemeter-client-tls\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.621854 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.621832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfxg\" (UniqueName: \"kubernetes.io/projected/680d50cc-d6f5-43de-955e-c286d1086113-kube-api-access-scfxg\") pod \"telemeter-client-5dff785474-x9xnb\" (UID: \"680d50cc-d6f5-43de-955e-c286d1086113\") " pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.638143 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.638112    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"60b1733dbcd6a4b71d1a486e3dc258f2f0f63e00780bf4018866fedc14a2dd70"}
Jul 24 13:10:44.638298 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.638193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"0cd31fd481d58e76b9c2a6a3cea18157a851956b8d5d110543b7bc359af20665"}
Jul 24 13:10:44.638298 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.638238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"de582c9b98f1052c084171460197b1ea1a3dd3f7092d969ddab75ed1955993e3"}
Jul 24 13:10:44.639660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.639628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt" event={"ID":"8ccc7612-b377-4303-903f-93b61d3d3725","Type":"ContainerStarted","Data":"a14e77c2682790536bc049d25d90dbf01906dda33d37f324893856ab5f724a8f"}
Jul 24 13:10:44.713718 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.713683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.716490 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.716466    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/731e5b78-6d8a-4388-9319-ccee65fc8bfc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dtkv6\" (UID: \"731e5b78-6d8a-4388-9319-ccee65fc8bfc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.788491 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.788463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb"
Jul 24 13:10:44.915279 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.915078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:44.967114 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:44.967086    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dff785474-x9xnb"]
Jul 24 13:10:44.971427 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:44.971394    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680d50cc_d6f5_43de_955e_c286d1086113.slice/crio-ad698d0a8482aa08f33e134dd7651fe678bc6f521a1b9f830de9e64b33fd46b9 WatchSource:0}: Error finding container ad698d0a8482aa08f33e134dd7651fe678bc6f521a1b9f830de9e64b33fd46b9: Status 404 returned error can't find the container with id ad698d0a8482aa08f33e134dd7651fe678bc6f521a1b9f830de9e64b33fd46b9
Jul 24 13:10:45.062916 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.062759    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"]
Jul 24 13:10:45.517723 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.515530    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:10:45.519156 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.519126    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.521962 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.521839    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 13:10:45.522205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.521848    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ca273uj2tjo7n\""
Jul 24 13:10:45.525807 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.525785    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 13:10:45.525992 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.525894    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 13:10:45.526098 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.526050    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 13:10:45.526185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.526161    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 13:10:45.526296 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.526253    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 13:10:45.526643 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.526619    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 13:10:45.526985 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.526964    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pcbwl\""
Jul 24 13:10:45.527281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.527133    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 13:10:45.527360 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.527307    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 13:10:45.527567 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.527551    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 13:10:45.527795 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.527772    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 13:10:45.527959 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.527923    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 13:10:45.529276 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.529247    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 13:10:45.532833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.532804    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 13:10:45.543861 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.543829    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:10:45.622976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.622953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754v8\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-kube-api-access-754v8\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623079 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.622996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623079 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623250    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623572    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.623674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.623652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.644185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.644157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6" event={"ID":"731e5b78-6d8a-4388-9319-ccee65fc8bfc","Type":"ContainerStarted","Data":"52537149d15e11f0a91886fe7bbdcaac6a28bb9c8d69abdcf634a40f7ddd843e"}
Jul 24 13:10:45.648764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.648734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"06040434d3b1ffddd203bef3c69d491803dd37284310687961f719f5493e958a"}
Jul 24 13:10:45.648907 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.648805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"2776cae05aa4326dbf5f68b4ad9bee20c9881a22c1d435eb4c52b38eb42e163c"}
Jul 24 13:10:45.648907 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.648846    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" event={"ID":"3783e967-19bc-4305-857b-776834ea1d54","Type":"ContainerStarted","Data":"f6ed4c09fa8c9a2a1c3c88aabb046cb426686ec27ff54fb5421849838f17579e"}
Jul 24 13:10:45.649059 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.648971    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:45.649965 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.649944    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb" event={"ID":"680d50cc-d6f5-43de-955e-c286d1086113","Type":"ContainerStarted","Data":"ad698d0a8482aa08f33e134dd7651fe678bc6f521a1b9f830de9e64b33fd46b9"}
Jul 24 13:10:45.676297 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.676244    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf" podStartSLOduration=1.985786863 podStartE2EDuration="4.676223313s" totalImagesPullingTime="2.69043645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:45.675019612 +0000 UTC m=+64.899502941" watchObservedRunningTime="2026-07-24 13:10:45.676223313 +0000 UTC m=+64.900706641"
Jul 24 13:10:45.724117 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754v8\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-kube-api-access-754v8\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.724559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724691    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724701    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724773    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.725108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.724834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.727150 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.726137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.728413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.728390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.729953 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.729910    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.730893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.731128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.731575    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.731638    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732636 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.732555    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732636 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.732596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559a7ec5-0658-424b-978b-60d599c48a6f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.732787 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.732769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-web-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.733142 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.733122    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.733986 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.733908    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.735608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.734325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/559a7ec5-0658-424b-978b-60d599c48a6f-config-out\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.735608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.734612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.735608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.735147    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.735608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.735561    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.736339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.736293    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.739174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.739134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-754v8\" (UniqueName: \"kubernetes.io/projected/559a7ec5-0658-424b-978b-60d599c48a6f-kube-api-access-754v8\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.739968 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.739899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/559a7ec5-0658-424b-978b-60d599c48a6f-config\") pod \"prometheus-k8s-0\" (UID: \"559a7ec5-0658-424b-978b-60d599c48a6f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:45.832322 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:45.832057    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:46.035846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.035805    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:10:46.040621 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:46.038100    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559a7ec5_0658_424b_978b_60d599c48a6f.slice/crio-a65c6e3739c893bb1a9a20aea72add5a27d1c35eee6dbb8904d37433582e723e WatchSource:0}: Error finding container a65c6e3739c893bb1a9a20aea72add5a27d1c35eee6dbb8904d37433582e723e: Status 404 returned error can't find the container with id a65c6e3739c893bb1a9a20aea72add5a27d1c35eee6dbb8904d37433582e723e
Jul 24 13:10:46.656119 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.656090    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt" event={"ID":"8ccc7612-b377-4303-903f-93b61d3d3725","Type":"ContainerStarted","Data":"b5d34df579f823d9bdfd13675b96ab9271bdd8ac8e8930d1e2157d51654c7ca8"}
Jul 24 13:10:46.658026 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.657919    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"a65c6e3739c893bb1a9a20aea72add5a27d1c35eee6dbb8904d37433582e723e"}
Jul 24 13:10:46.675490 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.675436    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt" podStartSLOduration=2.158304752 podStartE2EDuration="3.675421421s" totalImagesPullingTime="1.517116669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:46.674195641 +0000 UTC m=+65.898678969" watchObservedRunningTime="2026-07-24 13:10:46.675421421 +0000 UTC m=+65.899904748"
Jul 24 13:10:46.937916 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.937820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:46.938362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.937921    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:46.940272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.940250    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:10:46.940414 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.940297    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:10:46.950842 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.950815    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150f7de7-ecae-42c2-8942-dde8cc5c3203-metrics-certs\") pod \"network-metrics-daemon-rsqdv\" (UID: \"150f7de7-ecae-42c2-8942-dde8cc5c3203\") " pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:46.951068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.951039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ec6639f-9950-4bd0-85d5-a3542940b6aa-original-pull-secret\") pod \"global-pull-secret-syncer-5mp7r\" (UID: \"2ec6639f-9950-4bd0-85d5-a3542940b6aa\") " pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:46.951756 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:46.951740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5mp7r"
Jul 24 13:10:47.039060 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.039018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:47.041444 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.041416    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 13:10:47.051580 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.051560    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 13:10:47.063536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.063510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnggf\" (UniqueName: \"kubernetes.io/projected/1d80860e-1272-4cc4-8997-69b40184e1ca-kube-api-access-pnggf\") pod \"network-check-target-g75k9\" (UID: \"1d80860e-1272-4cc4-8997-69b40184e1ca\") " pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:47.250284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.250037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rh25q\""
Jul 24 13:10:47.259772 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.259538    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsqdv"
Jul 24 13:10:47.279598 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.279568    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5mp7r"]
Jul 24 13:10:47.283095 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:47.283056    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec6639f_9950_4bd0_85d5_a3542940b6aa.slice/crio-b6c5b6a77b7f8b3989e520fefc2861aae5f6a21fb1705b364fb43b61bca7ce4f WatchSource:0}: Error finding container b6c5b6a77b7f8b3989e520fefc2861aae5f6a21fb1705b364fb43b61bca7ce4f: Status 404 returned error can't find the container with id b6c5b6a77b7f8b3989e520fefc2861aae5f6a21fb1705b364fb43b61bca7ce4f
Jul 24 13:10:47.347948 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.347896    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dqwnd\""
Jul 24 13:10:47.355828 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.355807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:47.413101 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.413021    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsqdv"]
Jul 24 13:10:47.419199 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:47.419163    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150f7de7_ecae_42c2_8942_dde8cc5c3203.slice/crio-2a75aff2ccdde179072b5b398c979a2826cab0337e390bb168420b5ee4b9cd2e WatchSource:0}: Error finding container 2a75aff2ccdde179072b5b398c979a2826cab0337e390bb168420b5ee4b9cd2e: Status 404 returned error can't find the container with id 2a75aff2ccdde179072b5b398c979a2826cab0337e390bb168420b5ee4b9cd2e
Jul 24 13:10:47.503368 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.503333    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g75k9"]
Jul 24 13:10:47.508200 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:10:47.508178    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d80860e_1272_4cc4_8997_69b40184e1ca.slice/crio-76ccad0bd054595f6b3e82364fae855ef06681942038e2bae8d39489731fe1ed WatchSource:0}: Error finding container 76ccad0bd054595f6b3e82364fae855ef06681942038e2bae8d39489731fe1ed: Status 404 returned error can't find the container with id 76ccad0bd054595f6b3e82364fae855ef06681942038e2bae8d39489731fe1ed
Jul 24 13:10:47.664007 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.663957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6" event={"ID":"731e5b78-6d8a-4388-9319-ccee65fc8bfc","Type":"ContainerStarted","Data":"b00fa5e1a16f209a11494863e2b527b2f87b0f4a372638b878026f3b1b4e2be4"}
Jul 24 13:10:47.664206 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.664110    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:47.665369 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.665345    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g75k9" event={"ID":"1d80860e-1272-4cc4-8997-69b40184e1ca","Type":"ContainerStarted","Data":"76ccad0bd054595f6b3e82364fae855ef06681942038e2bae8d39489731fe1ed"}
Jul 24 13:10:47.666863 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.666841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsqdv" event={"ID":"150f7de7-ecae-42c2-8942-dde8cc5c3203","Type":"ContainerStarted","Data":"2a75aff2ccdde179072b5b398c979a2826cab0337e390bb168420b5ee4b9cd2e"}
Jul 24 13:10:47.668711 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.668687    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb" event={"ID":"680d50cc-d6f5-43de-955e-c286d1086113","Type":"ContainerStarted","Data":"6596cf728a00cb5880c613f188b3381f7fe0bb837bdf93da55e45b6564b6cbd4"}
Jul 24 13:10:47.670530 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.670511    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6"
Jul 24 13:10:47.670752 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.670733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5mp7r" event={"ID":"2ec6639f-9950-4bd0-85d5-a3542940b6aa","Type":"ContainerStarted","Data":"b6c5b6a77b7f8b3989e520fefc2861aae5f6a21fb1705b364fb43b61bca7ce4f"}
Jul 24 13:10:47.698204 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:47.698147    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dtkv6" podStartSLOduration=3.152718698 podStartE2EDuration="4.698121419s" totalImagesPullingTime="1.545402721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:47.680271455 +0000 UTC m=+66.904754786" watchObservedRunningTime="2026-07-24 13:10:47.698121419 +0000 UTC m=+66.922604750"
Jul 24 13:10:49.685309 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.685195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb" event={"ID":"680d50cc-d6f5-43de-955e-c286d1086113","Type":"ContainerStarted","Data":"7d17b73f33f8d57c81873469d85c17884cf91a9f8a959f5fde7f3eb58020955c"}
Jul 24 13:10:49.685309 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.685263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb" event={"ID":"680d50cc-d6f5-43de-955e-c286d1086113","Type":"ContainerStarted","Data":"f5294ec8726f4c2d85331274c87e19c1a7e895077da859df2b2acf61efe8589b"}
Jul 24 13:10:49.687123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.687090    2556 generic.go:356] "Generic (PLEG): container finished" podID="559a7ec5-0658-424b-978b-60d599c48a6f" containerID="77513b4057a969c29e2892823fc56eeb077e91ce4e4faec08a930f9006bcee79" exitCode=0
Jul 24 13:10:49.687252 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.687175    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerDied","Data":"77513b4057a969c29e2892823fc56eeb077e91ce4e4faec08a930f9006bcee79"}
Jul 24 13:10:49.690282 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.689874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsqdv" event={"ID":"150f7de7-ecae-42c2-8942-dde8cc5c3203","Type":"ContainerStarted","Data":"06965c0cc92cc87ca3acb73374e82819031e92f083417b0fb02a7dffcee999e5"}
Jul 24 13:10:49.690282 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.689914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsqdv" event={"ID":"150f7de7-ecae-42c2-8942-dde8cc5c3203","Type":"ContainerStarted","Data":"6baa097123057bc9c63fe8ed8759bba88894c05fd129c958dd705dd2339e94ad"}
Jul 24 13:10:49.707004 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.706781    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dff785474-x9xnb" podStartSLOduration=2.3160905 podStartE2EDuration="5.706765989s" totalImagesPullingTime="3.390675489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:49.705331684 +0000 UTC m=+68.929815012" watchObservedRunningTime="2026-07-24 13:10:49.706765989 +0000 UTC m=+68.931249316"
Jul 24 13:10:49.720886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:49.720834    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rsqdv" podStartSLOduration=67.361301427 podStartE2EDuration="1m8.720815839s" totalImagesPullingTime="1.359514412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:49.720783656 +0000 UTC m=+68.945266982" watchObservedRunningTime="2026-07-24 13:10:49.720815839 +0000 UTC m=+68.945299168"
Jul 24 13:10:51.665823 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:51.665789    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fd9ff656c-f9tpf"
Jul 24 13:10:54.719346 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.719310    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5mp7r" event={"ID":"2ec6639f-9950-4bd0-85d5-a3542940b6aa","Type":"ContainerStarted","Data":"6049799879e7f93500446e40abc91660f85daaeba2ffe970a79e87eeeec46f30"}
Jul 24 13:10:54.722187 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.722165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"c12eecd00928784568d17a0d1378e44bea3bdced3ae1161028c1c32b90d7b8ee"}
Jul 24 13:10:54.722347 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.722336    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"b94ea12645b18cc3c5313c4994e18614330f2b792f7af76b104ea241c27e4ba3"}
Jul 24 13:10:54.723896 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.723874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g75k9" event={"ID":"1d80860e-1272-4cc4-8997-69b40184e1ca","Type":"ContainerStarted","Data":"4b2b08bed9b553e2209c86aff7d1b9fefb26d321eb8ad3cae9f557fdd0947cbe"}
Jul 24 13:10:54.724261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.724247    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:10:54.734037 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.733997    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5mp7r" podStartSLOduration=66.918236909 podStartE2EDuration="1m13.733983748s" totalImagesPullingTime="6.815746839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:54.732557076 +0000 UTC m=+73.957040400" watchObservedRunningTime="2026-07-24 13:10:54.733983748 +0000 UTC m=+73.958467074"
Jul 24 13:10:54.746090 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:54.746033    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g75k9" podStartSLOduration=67.166717738 podStartE2EDuration="1m13.746013732s" totalImagesPullingTime="6.579295994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:54.745197213 +0000 UTC m=+73.969680542" watchObservedRunningTime="2026-07-24 13:10:54.746013732 +0000 UTC m=+73.970497062"
Jul 24 13:10:55.731366 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.731328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"b7570b774aaf0caa6d709c7fe9de07a1eac5149089a0d5e938def03e61992af7"}
Jul 24 13:10:55.731869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.731405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"29b74ec454b116b175115e2792f5b6a9b4c4e43cc81a41b206a1dcfccabde94e"}
Jul 24 13:10:55.731869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.731444    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"eb7d0c43063c584d76076e202535610248314004fabc2e4df9339ac977e5e0e2"}
Jul 24 13:10:55.731869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.731485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"559a7ec5-0658-424b-978b-60d599c48a6f","Type":"ContainerStarted","Data":"0f279c8e947ea7810fdacfaca31a5c08aca2d2e9241cba318c176b134af7ec8d"}
Jul 24 13:10:55.759533 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.759458    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.587783092 podStartE2EDuration="10.759437677s" totalImagesPullingTime="7.146761227s" totalInitContainerRuntime="24.893358ms" isStatefulPod=false podCreationTimestamp="2026-07-24 13:10:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:10:55.757648222 +0000 UTC m=+74.982131551" watchObservedRunningTime="2026-07-24 13:10:55.759437677 +0000 UTC m=+74.983921003"
Jul 24 13:10:55.832361 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:55.832334    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:10:59.708790 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:10:59.708734    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvjrb_1c6ed8d2-0deb-404f-ab68-6972a2ec9d03/serve-healthcheck-canary/0.log"
Jul 24 13:11:03.955415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:03.955298    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:11:03.955415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:03.955356    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:11:23.960712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:23.960679    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:11:23.964500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:23.964481    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cdf6bbc7c-6qfrt"
Jul 24 13:11:26.738515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:26.738481    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g75k9"
Jul 24 13:11:45.832868 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:45.832829    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:11:45.852880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:45.852856    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:11:45.936437 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:11:45.936412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:13:33.246514 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.246479    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"]
Jul 24 13:13:33.248729 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.248712    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.251771 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.251745    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:13:33.251898 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.251776    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-w2r4n\""
Jul 24 13:13:33.251898 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.251745    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 13:13:33.262574 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.262548    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"]
Jul 24 13:13:33.296634 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.296609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e838ad2c-fb30-4bca-850f-4d34636c4d31-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.296770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.296663    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v48w\" (UniqueName: \"kubernetes.io/projected/e838ad2c-fb30-4bca-850f-4d34636c4d31-kube-api-access-8v48w\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.397785 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.397757    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e838ad2c-fb30-4bca-850f-4d34636c4d31-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.398008 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.397830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v48w\" (UniqueName: \"kubernetes.io/projected/e838ad2c-fb30-4bca-850f-4d34636c4d31-kube-api-access-8v48w\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.398226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.398203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e838ad2c-fb30-4bca-850f-4d34636c4d31-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.406441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.406419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v48w\" (UniqueName: \"kubernetes.io/projected/e838ad2c-fb30-4bca-850f-4d34636c4d31-kube-api-access-8v48w\") pod \"openshift-lws-operator-fd8ccff4c-zwtlp\" (UID: \"e838ad2c-fb30-4bca-850f-4d34636c4d31\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.576040 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.576010    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"
Jul 24 13:13:33.702607 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:33.702583    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp"]
Jul 24 13:13:33.705239 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:13:33.705211    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode838ad2c_fb30_4bca_850f_4d34636c4d31.slice/crio-1a78d2517d0bcd824ffe0e001f8047c1b3200a3dedb79eba0b70fa6391c93da9 WatchSource:0}: Error finding container 1a78d2517d0bcd824ffe0e001f8047c1b3200a3dedb79eba0b70fa6391c93da9: Status 404 returned error can't find the container with id 1a78d2517d0bcd824ffe0e001f8047c1b3200a3dedb79eba0b70fa6391c93da9
Jul 24 13:13:34.314648 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.314614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp" event={"ID":"e838ad2c-fb30-4bca-850f-4d34636c4d31","Type":"ContainerStarted","Data":"1a78d2517d0bcd824ffe0e001f8047c1b3200a3dedb79eba0b70fa6391c93da9"}
Jul 24 13:13:34.334306 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.334277    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mw5rf"]
Jul 24 13:13:34.337437 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.337422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.339660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.339625    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 13:13:34.339779 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.339683    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 13:13:34.339779 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.339737    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-58q4m\""
Jul 24 13:13:34.347972 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.347950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mw5rf"]
Jul 24 13:13:34.405900 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.405873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-bound-sa-token\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.406049 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.405947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslsd\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-kube-api-access-qslsd\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.506951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.506902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslsd\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-kube-api-access-qslsd\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.507073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.507006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-bound-sa-token\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.516271 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.516240    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-bound-sa-token\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.516372 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.516307    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslsd\" (UniqueName: \"kubernetes.io/projected/a6d0631c-8e24-428f-89ec-6d29ddc98127-kube-api-access-qslsd\") pod \"cert-manager-55b4d954d5-mw5rf\" (UID: \"a6d0631c-8e24-428f-89ec-6d29ddc98127\") " pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.646710 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.646631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mw5rf"
Jul 24 13:13:34.775571 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:34.775523    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mw5rf"]
Jul 24 13:13:34.778130 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:13:34.778103    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6d0631c_8e24_428f_89ec_6d29ddc98127.slice/crio-ce1346c0c9819238dd665d761785ff2a438a8183a2738b5c316d8eb1bd876265 WatchSource:0}: Error finding container ce1346c0c9819238dd665d761785ff2a438a8183a2738b5c316d8eb1bd876265: Status 404 returned error can't find the container with id ce1346c0c9819238dd665d761785ff2a438a8183a2738b5c316d8eb1bd876265
Jul 24 13:13:35.321667 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:35.321631    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mw5rf" event={"ID":"a6d0631c-8e24-428f-89ec-6d29ddc98127","Type":"ContainerStarted","Data":"ce1346c0c9819238dd665d761785ff2a438a8183a2738b5c316d8eb1bd876265"}
Jul 24 13:13:37.332976 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:37.332850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp" event={"ID":"e838ad2c-fb30-4bca-850f-4d34636c4d31","Type":"ContainerStarted","Data":"2f245344381234bc87e2d3b5084a97077607851c4753a90973a629e8c7345b54"}
Jul 24 13:13:37.373518 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:37.372732    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zwtlp" podStartSLOduration=1.28785548 podStartE2EDuration="4.372708114s" totalImagesPullingTime="3.084852634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:13:37.371976489 +0000 UTC m=+236.596459819" watchObservedRunningTime="2026-07-24 13:13:37.372708114 +0000 UTC m=+236.597191442"
Jul 24 13:13:40.345972 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:40.345916    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mw5rf" event={"ID":"a6d0631c-8e24-428f-89ec-6d29ddc98127","Type":"ContainerStarted","Data":"27c44c55f437ba2dc7749d50b54c68f47cf7277e0d271535e9d596a004d8277f"}
Jul 24 13:13:40.362553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:40.362508    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mw5rf" podStartSLOduration=1.835424188 podStartE2EDuration="6.36249484s" totalImagesPullingTime="4.527070652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:13:40.361078233 +0000 UTC m=+239.585561560" watchObservedRunningTime="2026-07-24 13:13:40.36249484 +0000 UTC m=+239.586978166"
Jul 24 13:13:55.115949 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.115896    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"]
Jul 24 13:13:55.121822 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.121806    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.126616 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.126593    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 13:13:55.126805 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.126626    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 13:13:55.126805 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.126652    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 13:13:55.126805 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.126702    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-84bfr\""
Jul 24 13:13:55.127002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.126990    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 13:13:55.148713 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.148690    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"]
Jul 24 13:13:55.283583 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.283543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpc2d\" (UniqueName: \"kubernetes.io/projected/b007d022-84a4-4daa-94e5-d27bf8496da2-kube-api-access-lpc2d\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.283834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.283611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.283834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.283658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-tmp\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.283834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.283680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.283834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.283703    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-manifests\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.384917    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpc2d\" (UniqueName: \"kubernetes.io/projected/b007d022-84a4-4daa-94e5-d27bf8496da2-kube-api-access-lpc2d\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.385020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385222 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.385053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-tmp\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385222 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.385081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385222 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.385103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-manifests\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.385468 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.385445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-manifests\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.387465 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.387447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b007d022-84a4-4daa-94e5-d27bf8496da2-tmp\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.387588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.387572    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.387705 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.387689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b007d022-84a4-4daa-94e5-d27bf8496da2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.393198 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.393178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpc2d\" (UniqueName: \"kubernetes.io/projected/b007d022-84a4-4daa-94e5-d27bf8496da2-kube-api-access-lpc2d\") pod \"opendatahub-operator-controller-manager-dbfd75c57-cmwh7\" (UID: \"b007d022-84a4-4daa-94e5-d27bf8496da2\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.432035 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.432001    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:13:55.580824 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:55.580790    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"]
Jul 24 13:13:55.584768 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:13:55.584744    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007d022_84a4_4daa_94e5_d27bf8496da2.slice/crio-950ddcad3dc745f00eecca3ae243bc08b74f678791a9d5328c785e631f761bc7 WatchSource:0}: Error finding container 950ddcad3dc745f00eecca3ae243bc08b74f678791a9d5328c785e631f761bc7: Status 404 returned error can't find the container with id 950ddcad3dc745f00eecca3ae243bc08b74f678791a9d5328c785e631f761bc7
Jul 24 13:13:56.405573 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:56.405538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7" event={"ID":"b007d022-84a4-4daa-94e5-d27bf8496da2","Type":"ContainerStarted","Data":"950ddcad3dc745f00eecca3ae243bc08b74f678791a9d5328c785e631f761bc7"}
Jul 24 13:13:59.422731 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:13:59.422693    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7" event={"ID":"b007d022-84a4-4daa-94e5-d27bf8496da2","Type":"ContainerStarted","Data":"d9d9b6703f80ca44eb9043620fa0dddd12f697669ecdf3a3034cd65ec7bf2bc1"}
Jul 24 13:14:00.428906 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:00.428817    2556 generic.go:356] "Generic (PLEG): container finished" podID="b007d022-84a4-4daa-94e5-d27bf8496da2" containerID="d9d9b6703f80ca44eb9043620fa0dddd12f697669ecdf3a3034cd65ec7bf2bc1" exitCode=0
Jul 24 13:14:00.428906 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:00.428881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7" event={"ID":"b007d022-84a4-4daa-94e5-d27bf8496da2","Type":"ContainerDied","Data":"d9d9b6703f80ca44eb9043620fa0dddd12f697669ecdf3a3034cd65ec7bf2bc1"}
Jul 24 13:14:01.435786 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:01.435676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7" event={"ID":"b007d022-84a4-4daa-94e5-d27bf8496da2","Type":"ContainerStarted","Data":"3d05ce2ef668b3f39d5f4bd79b17fab0d5d35d867e5fa36e0d758e8a30a1e43f"}
Jul 24 13:14:01.435786 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:01.435750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:14:01.456320 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:01.456255    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7" podStartSLOduration=1.321726361 podStartE2EDuration="6.456234427s" totalImagesPullingTime="4.065216536s" totalInitContainerRuntime="1.06929153s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:13:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:14:01.455012359 +0000 UTC m=+260.679495687" watchObservedRunningTime="2026-07-24 13:14:01.456234427 +0000 UTC m=+260.680717755"
Jul 24 13:14:12.444569 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:12.444533    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbfd75c57-cmwh7"
Jul 24 13:14:13.062406 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.062364    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"]
Jul 24 13:14:13.072029 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.071999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.073174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.073145    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"]
Jul 24 13:14:13.074322 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.074300    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 13:14:13.074415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.074374    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 13:14:13.075123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.075101    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 13:14:13.075220 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.075133    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rcphn\""
Jul 24 13:14:13.129429 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.129402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.129576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.129438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghb2m\" (UniqueName: \"kubernetes.io/projected/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-kube-api-access-ghb2m\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.129576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.129473    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-metrics-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.129576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.129513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-manager-config\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.230099 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.230068    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-manager-config\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.230273 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.230134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.230273 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.230154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghb2m\" (UniqueName: \"kubernetes.io/projected/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-kube-api-access-ghb2m\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.230273 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.230187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-metrics-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.230874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.230800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-manager-config\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.232680 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.232660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-metrics-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.232775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.232681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-cert\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.240718 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.240690    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghb2m\" (UniqueName: \"kubernetes.io/projected/8bafc74a-e0ef-4a25-bca8-cda204c4f50d-kube-api-access-ghb2m\") pod \"lws-controller-manager-5d6cbddddd-7nz8b\" (UID: \"8bafc74a-e0ef-4a25-bca8-cda204c4f50d\") " pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.382260 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.382170    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:13.512458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:13.512427    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"]
Jul 24 13:14:13.516577 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:14:13.516552    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bafc74a_e0ef_4a25_bca8_cda204c4f50d.slice/crio-5786f3ff271478339bc4c9b47bf873b83c0a9beba4daf96fd2c80c7a11c86e30 WatchSource:0}: Error finding container 5786f3ff271478339bc4c9b47bf873b83c0a9beba4daf96fd2c80c7a11c86e30: Status 404 returned error can't find the container with id 5786f3ff271478339bc4c9b47bf873b83c0a9beba4daf96fd2c80c7a11c86e30
Jul 24 13:14:14.494660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:14.494621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b" event={"ID":"8bafc74a-e0ef-4a25-bca8-cda204c4f50d","Type":"ContainerStarted","Data":"5786f3ff271478339bc4c9b47bf873b83c0a9beba4daf96fd2c80c7a11c86e30"}
Jul 24 13:14:15.310190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.310109    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"]
Jul 24 13:14:15.313190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.313175    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.318671 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.318648    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 13:14:15.319838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.319805    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 13:14:15.320294 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.320276    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 13:14:15.320459 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.320446    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 13:14:15.328367 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.328351    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-twt4h\""
Jul 24 13:14:15.334354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.334334    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"]
Jul 24 13:14:15.347779 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.347757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2018df84-5adc-4461-9096-f6f6841e6f19-tmp\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.347900 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.347788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqzq\" (UniqueName: \"kubernetes.io/projected/2018df84-5adc-4461-9096-f6f6841e6f19-kube-api-access-cbqzq\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.347900 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.347862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2018df84-5adc-4461-9096-f6f6841e6f19-tls-certs\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.448951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.448892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2018df84-5adc-4461-9096-f6f6841e6f19-tmp\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.448951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.448926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqzq\" (UniqueName: \"kubernetes.io/projected/2018df84-5adc-4461-9096-f6f6841e6f19-kube-api-access-cbqzq\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.449159 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.448978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2018df84-5adc-4461-9096-f6f6841e6f19-tls-certs\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.451255 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.451233    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2018df84-5adc-4461-9096-f6f6841e6f19-tmp\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.451454 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.451437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2018df84-5adc-4461-9096-f6f6841e6f19-tls-certs\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.472175 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.472148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqzq\" (UniqueName: \"kubernetes.io/projected/2018df84-5adc-4461-9096-f6f6841e6f19-kube-api-access-cbqzq\") pod \"kube-auth-proxy-76dcd6cb6b-xzl65\" (UID: \"2018df84-5adc-4461-9096-f6f6841e6f19\") " pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.500602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.500575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b" event={"ID":"8bafc74a-e0ef-4a25-bca8-cda204c4f50d","Type":"ContainerStarted","Data":"b0b2232caa9e56d8e961d557c16922c60d6bf61b6435d5159ac12df617033254"}
Jul 24 13:14:15.500703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.500632    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:15.623184 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.623110    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"
Jul 24 13:14:15.803017 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.802974    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b" podStartSLOduration=1.28056417 podStartE2EDuration="2.802948216s" totalImagesPullingTime="1.522384046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:14:15.540082225 +0000 UTC m=+274.764565551" watchObservedRunningTime="2026-07-24 13:14:15.802948216 +0000 UTC m=+275.027431544"
Jul 24 13:14:15.803189 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:15.803173    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65"]
Jul 24 13:14:15.805452 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:14:15.805428    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2018df84_5adc_4461_9096_f6f6841e6f19.slice/crio-1f998648f5cc730765ad49b3098c72e7884e7b32d23ce0af5f268d61454099cf WatchSource:0}: Error finding container 1f998648f5cc730765ad49b3098c72e7884e7b32d23ce0af5f268d61454099cf: Status 404 returned error can't find the container with id 1f998648f5cc730765ad49b3098c72e7884e7b32d23ce0af5f268d61454099cf
Jul 24 13:14:16.515106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:16.515059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65" event={"ID":"2018df84-5adc-4461-9096-f6f6841e6f19","Type":"ContainerStarted","Data":"1f998648f5cc730765ad49b3098c72e7884e7b32d23ce0af5f268d61454099cf"}
Jul 24 13:14:18.525272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:18.525240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65" event={"ID":"2018df84-5adc-4461-9096-f6f6841e6f19","Type":"ContainerStarted","Data":"62198952b98fea08e5b127f668ee76c92eb438c3fdc2893b40d25a386d10dc2e"}
Jul 24 13:14:18.544100 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:18.544058    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76dcd6cb6b-xzl65" podStartSLOduration=1.22217388 podStartE2EDuration="3.544043426s" totalImagesPullingTime="2.321869546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:14:18.542986751 +0000 UTC m=+277.767470079" watchObservedRunningTime="2026-07-24 13:14:18.544043426 +0000 UTC m=+277.768526781"
Jul 24 13:14:26.517876 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:26.517848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d6cbddddd-7nz8b"
Jul 24 13:14:41.133775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:41.133747    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:14:41.139810 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:41.139785    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 13:14:41.146886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:14:41.146855    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:15:55.803222 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.803133    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"]
Jul 24 13:15:55.805401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.805386    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:55.808546 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.808525    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 13:15:55.809285 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.809269    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 13:15:55.809370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.809273    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nv4vb\""
Jul 24 13:15:55.882334 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.882303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksl2z\" (UniqueName: \"kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z\") pod \"limitador-operator-controller-manager-859d7b78bd-x42g5\" (UID: \"46350d28-c2b0-4fce-b743-b373b874fef9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:55.899662 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.899636    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"]
Jul 24 13:15:55.982908 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:55.982876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksl2z\" (UniqueName: \"kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z\") pod \"limitador-operator-controller-manager-859d7b78bd-x42g5\" (UID: \"46350d28-c2b0-4fce-b743-b373b874fef9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:56.017980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.017947    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksl2z\" (UniqueName: \"kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z\") pod \"limitador-operator-controller-manager-859d7b78bd-x42g5\" (UID: \"46350d28-c2b0-4fce-b743-b373b874fef9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:56.115226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.115124    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:56.271141 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.271067    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"]
Jul 24 13:15:56.275764 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:15:56.275740    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46350d28_c2b0_4fce_b743_b373b874fef9.slice/crio-a73b751582f8a7644934ecc5064e2705777ab70b374db1c11c84a6f20949c7ab WatchSource:0}: Error finding container a73b751582f8a7644934ecc5064e2705777ab70b374db1c11c84a6f20949c7ab: Status 404 returned error can't find the container with id a73b751582f8a7644934ecc5064e2705777ab70b374db1c11c84a6f20949c7ab
Jul 24 13:15:56.277750 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.277732    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:15:56.689314 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.689278    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"]
Jul 24 13:15:56.692191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.692176    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.698721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.698603    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v9v7d\""
Jul 24 13:15:56.698721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.698652    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 13:15:56.698880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.698723    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 13:15:56.713182 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.713150    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"]
Jul 24 13:15:56.790088 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.790053    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.790275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.790106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.790275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.790173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kl94\" (UniqueName: \"kubernetes.io/projected/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-kube-api-access-6kl94\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.890789 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.890758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.891190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.890808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.891190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.890867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kl94\" (UniqueName: \"kubernetes.io/projected/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-kube-api-access-6kl94\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.891500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.891482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.893500 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.893483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.901310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.901289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kl94\" (UniqueName: \"kubernetes.io/projected/7ccb5e85-d10e-4dd5-a161-9cedccbf0aed-kube-api-access-6kl94\") pod \"kuadrant-console-plugin-66b5774985-xsqc2\" (UID: \"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:56.943547 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:56.943460    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5" event={"ID":"46350d28-c2b0-4fce-b743-b373b874fef9","Type":"ContainerStarted","Data":"a73b751582f8a7644934ecc5064e2705777ab70b374db1c11c84a6f20949c7ab"}
Jul 24 13:15:57.001203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:57.001174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"
Jul 24 13:15:57.271596 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:57.271567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2"]
Jul 24 13:15:57.374056 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:15:57.374022    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ccb5e85_d10e_4dd5_a161_9cedccbf0aed.slice/crio-72332f3314f01dbd3d13a81f3cc5532b9e8db7ea00f17e4d7e48722606ac97de WatchSource:0}: Error finding container 72332f3314f01dbd3d13a81f3cc5532b9e8db7ea00f17e4d7e48722606ac97de: Status 404 returned error can't find the container with id 72332f3314f01dbd3d13a81f3cc5532b9e8db7ea00f17e4d7e48722606ac97de
Jul 24 13:15:57.950558 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:57.950523    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2" event={"ID":"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed","Type":"ContainerStarted","Data":"72332f3314f01dbd3d13a81f3cc5532b9e8db7ea00f17e4d7e48722606ac97de"}
Jul 24 13:15:58.957828 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:58.957788    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5" event={"ID":"46350d28-c2b0-4fce-b743-b373b874fef9","Type":"ContainerStarted","Data":"a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae"}
Jul 24 13:15:58.958345 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:58.957890    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:15:58.979669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:15:58.979596    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5" podStartSLOduration=1.826385464 podStartE2EDuration="3.97958098s" totalImagesPullingTime="2.153195516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:15:58.97821983 +0000 UTC m=+378.202703159" watchObservedRunningTime="2026-07-24 13:15:58.97958098 +0000 UTC m=+378.204064307"
Jul 24 13:16:06.478789 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:06.478757    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"]
Jul 24 13:16:06.479331 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:06.479060    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" containerName="manager" containerID="cri-o://a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae" gracePeriod=2
Jul 24 13:16:06.483062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:06.482921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:06.483062 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:06.482921    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:06.485023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:06.484988    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"]
Jul 24 13:16:06.492749 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:06.492702    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:18.225396 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:18.225370    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:18.227710 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:18.227677    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:18.288590 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:18.288553    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksl2z\" (UniqueName: \"kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z\") pod \"46350d28-c2b0-4fce-b743-b373b874fef9\" (UID: \"46350d28-c2b0-4fce-b743-b373b874fef9\") "
Jul 24 13:16:18.290696 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:18.290676    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z" (OuterVolumeSpecName: "kube-api-access-ksl2z") pod "46350d28-c2b0-4fce-b743-b373b874fef9" (UID: "46350d28-c2b0-4fce-b743-b373b874fef9"). InnerVolumeSpecName "kube-api-access-ksl2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:16:18.389538 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:18.389450    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksl2z\" (UniqueName: \"kubernetes.io/projected/46350d28-c2b0-4fce-b743-b373b874fef9-kube-api-access-ksl2z\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:16:19.068162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.068123    2556 generic.go:356] "Generic (PLEG): container finished" podID="46350d28-c2b0-4fce-b743-b373b874fef9" containerID="a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae" exitCode=0
Jul 24 13:16:19.068162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.068165    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:19.068406 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.068222    2556 scope.go:117] "RemoveContainer" containerID="a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae"
Jul 24 13:16:19.069998 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.069958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2" event={"ID":"7ccb5e85-d10e-4dd5-a161-9cedccbf0aed","Type":"ContainerStarted","Data":"aeb6c33f65754891aa1e47f0b3843a1bd6916305bce5f70685800857a49603b7"}
Jul 24 13:16:19.070451 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:19.070429    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:19.072038 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:19.072015    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:19.077847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.077829    2556 scope.go:117] "RemoveContainer" containerID="a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae"
Jul 24 13:16:19.078121 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:19.078103    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae\": container with ID starting with a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae not found: ID does not exist" containerID="a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae"
Jul 24 13:16:19.078195 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.078136    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae"} err="failed to get container status \"a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae\": rpc error: code = NotFound desc = could not find container \"a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae\": container with ID starting with a84612a24e03f5bdae27768bea26d6671f2a4701ae0a447327a19a1c72a3e3ae not found: ID does not exist"
Jul 24 13:16:19.089904 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.089831    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xsqc2" podStartSLOduration=2.292401426 podStartE2EDuration="23.089817888s" totalImagesPullingTime="20.797416462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:16:19.088422595 +0000 UTC m=+398.312905922" watchObservedRunningTime="2026-07-24 13:16:19.089817888 +0000 UTC m=+398.314301219"
Jul 24 13:16:19.090048 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:19.090024    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x42g5\" is forbidden: User \"system:node:ip-10-0-136-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-14.ec2.internal' and this object" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x42g5"
Jul 24 13:16:19.345121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:19.345034    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" path="/var/lib/kubelet/pods/46350d28-c2b0-4fce-b743-b373b874fef9/volumes"
Jul 24 13:16:38.988268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:38.988231    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:38.988737 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:38.988544    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" containerName="manager"
Jul 24 13:16:38.988737 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:38.988556    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" containerName="manager"
Jul 24 13:16:38.988737 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:38.988611    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="46350d28-c2b0-4fce-b743-b373b874fef9" containerName="manager"
Jul 24 13:16:39.059490 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.059458    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:39.059652 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.059552    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.062006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.061983    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 13:16:39.084762 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.084735    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:39.171374 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.171345    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wws8q\" (UniqueName: \"kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.171555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.171382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.271969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.271867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.272120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.271968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wws8q\" (UniqueName: \"kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.272501 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.272480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.280469 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.280444    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wws8q\" (UniqueName: \"kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q\") pod \"limitador-limitador-64c94fd64f-qpgvc\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.370210 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.370182    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:39.513231 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.513085    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:39.514132 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:16:39.514107    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe8d39c_0bc3_40b6_9bb9_07c74bf0e964.slice/crio-a18c82044eb0cbf29c5e1d4c253dafa11fa7bb241e60da669165d6e3dee3a015 WatchSource:0}: Error finding container a18c82044eb0cbf29c5e1d4c253dafa11fa7bb241e60da669165d6e3dee3a015: Status 404 returned error can't find the container with id a18c82044eb0cbf29c5e1d4c253dafa11fa7bb241e60da669165d6e3dee3a015
Jul 24 13:16:39.801904 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.801872    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:39.806439 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.806416    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:39.808438 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.808417    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jfbtl\""
Jul 24 13:16:39.810982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.810962    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:39.978380 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:39.978342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlq5r\" (UniqueName: \"kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r\") pod \"authorino-58cd955675-49n7j\" (UID: \"93fdc096-1236-4b69-852f-4c65e15ba73e\") " pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:40.078842 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:40.078752    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlq5r\" (UniqueName: \"kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r\") pod \"authorino-58cd955675-49n7j\" (UID: \"93fdc096-1236-4b69-852f-4c65e15ba73e\") " pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:40.091576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:40.091549    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlq5r\" (UniqueName: \"kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r\") pod \"authorino-58cd955675-49n7j\" (UID: \"93fdc096-1236-4b69-852f-4c65e15ba73e\") " pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:40.116520 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:40.116486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:40.162681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:40.162598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" event={"ID":"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964","Type":"ContainerStarted","Data":"a18c82044eb0cbf29c5e1d4c253dafa11fa7bb241e60da669165d6e3dee3a015"}
Jul 24 13:16:40.281660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:40.281219    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:41.168922 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:41.168892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-49n7j" event={"ID":"93fdc096-1236-4b69-852f-4c65e15ba73e","Type":"ContainerStarted","Data":"0d03ef881c98f382a869bc1534c9a3bb599bd3749ee7f356e88d5e9fcaa4a66b"}
Jul 24 13:16:43.184281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:43.184251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" event={"ID":"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964","Type":"ContainerStarted","Data":"3f6aeec21441588f3de3218268ec1b72859fc8a6f8755ddcd03c27771e58ba94"}
Jul 24 13:16:43.184718 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:43.184388    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:43.204022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:43.203959    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" podStartSLOduration=2.385247416 podStartE2EDuration="5.203912105s" totalImagesPullingTime="2.818664689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:16:43.202372284 +0000 UTC m=+422.426855625" watchObservedRunningTime="2026-07-24 13:16:43.203912105 +0000 UTC m=+422.428395433"
Jul 24 13:16:43.883440 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:43.883394    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:45.194483 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.194448    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-49n7j" event={"ID":"93fdc096-1236-4b69-852f-4c65e15ba73e","Type":"ContainerStarted","Data":"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"}
Jul 24 13:16:45.195020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.194498    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-49n7j" podUID="93fdc096-1236-4b69-852f-4c65e15ba73e" containerName="authorino" containerID="cri-o://e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a" gracePeriod=30
Jul 24 13:16:45.209311 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.209271    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-49n7j" podStartSLOduration=2.346773978 podStartE2EDuration="6.209234728s" totalImagesPullingTime="3.86246075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:16:45.208047937 +0000 UTC m=+424.432531266" watchObservedRunningTime="2026-07-24 13:16:45.209234728 +0000 UTC m=+424.433718055"
Jul 24 13:16:45.434361 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.434339    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:45.527168 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.527147    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlq5r\" (UniqueName: \"kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r\") pod \"93fdc096-1236-4b69-852f-4c65e15ba73e\" (UID: \"93fdc096-1236-4b69-852f-4c65e15ba73e\") "
Jul 24 13:16:45.529274 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.529242    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r" (OuterVolumeSpecName: "kube-api-access-dlq5r") pod "93fdc096-1236-4b69-852f-4c65e15ba73e" (UID: "93fdc096-1236-4b69-852f-4c65e15ba73e"). InnerVolumeSpecName "kube-api-access-dlq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:16:45.627889 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:45.627840    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlq5r\" (UniqueName: \"kubernetes.io/projected/93fdc096-1236-4b69-852f-4c65e15ba73e-kube-api-access-dlq5r\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:16:46.199880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.199831    2556 generic.go:356] "Generic (PLEG): container finished" podID="93fdc096-1236-4b69-852f-4c65e15ba73e" containerID="e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a" exitCode=0
Jul 24 13:16:46.200362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.199897    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-49n7j"
Jul 24 13:16:46.200362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.199910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-49n7j" event={"ID":"93fdc096-1236-4b69-852f-4c65e15ba73e","Type":"ContainerDied","Data":"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"}
Jul 24 13:16:46.200362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.199975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-49n7j" event={"ID":"93fdc096-1236-4b69-852f-4c65e15ba73e","Type":"ContainerDied","Data":"0d03ef881c98f382a869bc1534c9a3bb599bd3749ee7f356e88d5e9fcaa4a66b"}
Jul 24 13:16:46.200362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.199998    2556 scope.go:117] "RemoveContainer" containerID="e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"
Jul 24 13:16:46.208884 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.208863    2556 scope.go:117] "RemoveContainer" containerID="e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"
Jul 24 13:16:46.209180 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:16:46.209148    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a\": container with ID starting with e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a not found: ID does not exist" containerID="e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"
Jul 24 13:16:46.209265 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.209190    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a"} err="failed to get container status \"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a\": rpc error: code = NotFound desc = could not find container \"e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a\": container with ID starting with e19f690e94088791968774472ecf3510d47c264bb803787cbe75fef779a79f9a not found: ID does not exist"
Jul 24 13:16:46.221365 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.221340    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:46.225885 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:46.225866    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-49n7j"]
Jul 24 13:16:47.342359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:47.342305    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93fdc096-1236-4b69-852f-4c65e15ba73e" path="/var/lib/kubelet/pods/93fdc096-1236-4b69-852f-4c65e15ba73e/volumes"
Jul 24 13:16:53.714477 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:53.714440    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:53.714924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:53.714629    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" podUID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" containerName="limitador" containerID="cri-o://3f6aeec21441588f3de3218268ec1b72859fc8a6f8755ddcd03c27771e58ba94" gracePeriod=30
Jul 24 13:16:53.715327 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:53.715300    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:54.236257 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.236231    2556 generic.go:356] "Generic (PLEG): container finished" podID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" containerID="3f6aeec21441588f3de3218268ec1b72859fc8a6f8755ddcd03c27771e58ba94" exitCode=0
Jul 24 13:16:54.236393 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.236291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" event={"ID":"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964","Type":"ContainerDied","Data":"3f6aeec21441588f3de3218268ec1b72859fc8a6f8755ddcd03c27771e58ba94"}
Jul 24 13:16:54.253298 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.253274    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:54.298173 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.298145    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file\") pod \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") "
Jul 24 13:16:54.298339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.298219    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wws8q\" (UniqueName: \"kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q\") pod \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\" (UID: \"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964\") "
Jul 24 13:16:54.298489 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.298467    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file" (OuterVolumeSpecName: "config-file") pod "0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" (UID: "0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:16:54.300336 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.300315    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q" (OuterVolumeSpecName: "kube-api-access-wws8q") pod "0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" (UID: "0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964"). InnerVolumeSpecName "kube-api-access-wws8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:16:54.399053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.399015    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-config-file\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:16:54.399053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:54.399046    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wws8q\" (UniqueName: \"kubernetes.io/projected/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964-kube-api-access-wws8q\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:16:55.242049 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.242020    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"
Jul 24 13:16:55.242049 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.242040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qpgvc" event={"ID":"0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964","Type":"ContainerDied","Data":"a18c82044eb0cbf29c5e1d4c253dafa11fa7bb241e60da669165d6e3dee3a015"}
Jul 24 13:16:55.242509 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.242095    2556 scope.go:117] "RemoveContainer" containerID="3f6aeec21441588f3de3218268ec1b72859fc8a6f8755ddcd03c27771e58ba94"
Jul 24 13:16:55.286711 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.286681    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:55.289259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.289240    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qpgvc"]
Jul 24 13:16:55.345252 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.341747    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" path="/var/lib/kubelet/pods/0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964/volumes"
Jul 24 13:16:55.643536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643505    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"]
Jul 24 13:16:55.643854 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643841    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" containerName="limitador"
Jul 24 13:16:55.643893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643856    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" containerName="limitador"
Jul 24 13:16:55.643893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643877    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93fdc096-1236-4b69-852f-4c65e15ba73e" containerName="authorino"
Jul 24 13:16:55.643893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643886    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="93fdc096-1236-4b69-852f-4c65e15ba73e" containerName="authorino"
Jul 24 13:16:55.644032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643979    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="93fdc096-1236-4b69-852f-4c65e15ba73e" containerName="authorino"
Jul 24 13:16:55.644032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.643988    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe8d39c-0bc3-40b6-9bb9-07c74bf0e964" containerName="limitador"
Jul 24 13:16:55.648287 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.648269    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.650702 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.650680    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-k7hzt\""
Jul 24 13:16:55.650844 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.650689    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:16:55.650844 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.650724    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 13:16:55.651301 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.651285    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 13:16:55.658715 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.658692    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"]
Jul 24 13:16:55.709226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.709193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-data\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.709351 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.709237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msk4p\" (UniqueName: \"kubernetes.io/projected/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-kube-api-access-msk4p\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.810536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.810500    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-data\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.810698 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.810543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msk4p\" (UniqueName: \"kubernetes.io/projected/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-kube-api-access-msk4p\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.810900 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.810882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-data\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.819310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.819280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msk4p\" (UniqueName: \"kubernetes.io/projected/a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a-kube-api-access-msk4p\") pod \"postgres-69fb4594cf-8pg9x\" (UID: \"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:55.959235 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:55.959153    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:16:56.089292 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:56.089270    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"]
Jul 24 13:16:56.091358 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:16:56.091335    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ca0f3c_581f_4fec_ac8f_b4a70ccc291a.slice/crio-5be1b7eb311edc562ac379be3f0bee5dfbeae8ef135f3625f81cd7289dac736e WatchSource:0}: Error finding container 5be1b7eb311edc562ac379be3f0bee5dfbeae8ef135f3625f81cd7289dac736e: Status 404 returned error can't find the container with id 5be1b7eb311edc562ac379be3f0bee5dfbeae8ef135f3625f81cd7289dac736e
Jul 24 13:16:56.247864 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:16:56.247765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x" event={"ID":"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a","Type":"ContainerStarted","Data":"5be1b7eb311edc562ac379be3f0bee5dfbeae8ef135f3625f81cd7289dac736e"}
Jul 24 13:17:01.273621 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:01.273593    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x" event={"ID":"a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a","Type":"ContainerStarted","Data":"687472d8eb9aed103b3b193e6dabcffe9c92cd0cacea62a392b9623d9bf10900"}
Jul 24 13:17:01.274087 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:01.273700    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:17:01.290244 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:01.290200    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x" podStartSLOduration=1.462355082 podStartE2EDuration="6.290184595s" totalImagesPullingTime="4.827829513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:17:01.29013746 +0000 UTC m=+440.514620789" watchObservedRunningTime="2026-07-24 13:17:01.290184595 +0000 UTC m=+440.514667900"
Jul 24 13:17:07.306960 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.306909    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8pg9x"
Jul 24 13:17:07.861582 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.861544    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:07.865532 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.865515    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:07.868136 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.868116    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jfbtl\""
Jul 24 13:17:07.872856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.872838    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:07.921840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:07.921808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmm5\" (UniqueName: \"kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5\") pod \"authorino-5d4b67c446-fxlxq\" (UID: \"f7345e5d-b124-4c99-a1bf-0f284473b105\") " pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:08.023155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.023119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmm5\" (UniqueName: \"kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5\") pod \"authorino-5d4b67c446-fxlxq\" (UID: \"f7345e5d-b124-4c99-a1bf-0f284473b105\") " pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:08.036820 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.036797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmm5\" (UniqueName: \"kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5\") pod \"authorino-5d4b67c446-fxlxq\" (UID: \"f7345e5d-b124-4c99-a1bf-0f284473b105\") " pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:08.048025 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.048000    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:08.048256 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.048244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:08.073986 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.073956    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-l6jqw"]
Jul 24 13:17:08.076466 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.076444    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.078968 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.078949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 13:17:08.084491 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.084472    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-l6jqw"]
Jul 24 13:17:08.120956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.120861    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-l6jqw"]
Jul 24 13:17:08.121212 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:17:08.121187    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cmkkg tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-cmkkg tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-l6jqw" podUID="db5767a2-0002-4431-9d87-1f7962ef27f0"
Jul 24 13:17:08.150392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.150094    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:17:08.153083 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.153061    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.163447 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.163420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:17:08.181763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.181733    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:08.184782 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:17:08.184758    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7345e5d_b124_4c99_a1bf_0f284473b105.slice/crio-a546278fca7e8bbaf33382d99f26b1bd718d85515a3a07d2c767f9f60eb8c1cc WatchSource:0}: Error finding container a546278fca7e8bbaf33382d99f26b1bd718d85515a3a07d2c767f9f60eb8c1cc: Status 404 returned error can't find the container with id a546278fca7e8bbaf33382d99f26b1bd718d85515a3a07d2c767f9f60eb8c1cc
Jul 24 13:17:08.225357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.225331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkkg\" (UniqueName: \"kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.225484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.225363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.225484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.225398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.225549 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.225494    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28j6x\" (UniqueName: \"kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.303321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.303294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.303472 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.303290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" event={"ID":"f7345e5d-b124-4c99-a1bf-0f284473b105","Type":"ContainerStarted","Data":"a546278fca7e8bbaf33382d99f26b1bd718d85515a3a07d2c767f9f60eb8c1cc"}
Jul 24 13:17:08.309309 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.309293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.326772 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.326747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkkg\" (UniqueName: \"kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.326869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.326779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.326869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.326814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.326982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.326925    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28j6x\" (UniqueName: \"kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.329297 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.329273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.329390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.329332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.334521 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.334501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28j6x\" (UniqueName: \"kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x\") pod \"authorino-7fdbf4595d-v42bw\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") " pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.334617 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.334587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkkg\" (UniqueName: \"kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg\") pod \"authorino-76877994-l6jqw\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") " pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:08.428185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.428095    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert\") pod \"db5767a2-0002-4431-9d87-1f7962ef27f0\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") "
Jul 24 13:17:08.428185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.428176    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmkkg\" (UniqueName: \"kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg\") pod \"db5767a2-0002-4431-9d87-1f7962ef27f0\" (UID: \"db5767a2-0002-4431-9d87-1f7962ef27f0\") "
Jul 24 13:17:08.430402 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.430377    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "db5767a2-0002-4431-9d87-1f7962ef27f0" (UID: "db5767a2-0002-4431-9d87-1f7962ef27f0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:17:08.430499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.430423    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg" (OuterVolumeSpecName: "kube-api-access-cmkkg") pod "db5767a2-0002-4431-9d87-1f7962ef27f0" (UID: "db5767a2-0002-4431-9d87-1f7962ef27f0"). InnerVolumeSpecName "kube-api-access-cmkkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:17:08.462716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.462689    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:17:08.529026 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.528998    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db5767a2-0002-4431-9d87-1f7962ef27f0-tls-cert\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:17:08.529026 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.529026    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkkg\" (UniqueName: \"kubernetes.io/projected/db5767a2-0002-4431-9d87-1f7962ef27f0-kube-api-access-cmkkg\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:17:08.589322 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:08.589294    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:17:08.591155 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:17:08.591125    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0aa0c6a_7b42_494a_8dd9_c20a1d3bf2f6.slice/crio-c1ec723e9b801205a2ec578243be27ddcb3187669fdf4fd86575a9294252ea31 WatchSource:0}: Error finding container c1ec723e9b801205a2ec578243be27ddcb3187669fdf4fd86575a9294252ea31: Status 404 returned error can't find the container with id c1ec723e9b801205a2ec578243be27ddcb3187669fdf4fd86575a9294252ea31
Jul 24 13:17:09.309543 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.309436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" event={"ID":"f7345e5d-b124-4c99-a1bf-0f284473b105","Type":"ContainerStarted","Data":"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"}
Jul 24 13:17:09.310043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.309547    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" podUID="f7345e5d-b124-4c99-a1bf-0f284473b105" containerName="authorino" containerID="cri-o://9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a" gracePeriod=30
Jul 24 13:17:09.312336 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.312299    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" event={"ID":"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6","Type":"ContainerStarted","Data":"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"}
Jul 24 13:17:09.312443 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.312347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" event={"ID":"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6","Type":"ContainerStarted","Data":"c1ec723e9b801205a2ec578243be27ddcb3187669fdf4fd86575a9294252ea31"}
Jul 24 13:17:09.312443 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.312384    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-l6jqw"
Jul 24 13:17:09.323376 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.323323    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" podStartSLOduration=1.920983216 podStartE2EDuration="2.323303895s" totalImagesPullingTime="402.320679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:17:09.322643004 +0000 UTC m=+448.547126336" watchObservedRunningTime="2026-07-24 13:17:09.323303895 +0000 UTC m=+448.547787222"
Jul 24 13:17:09.342270 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.342201    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" podStartSLOduration=0.732992781 podStartE2EDuration="1.342180426s" totalImagesPullingTime="609.187645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:17:09.340536394 +0000 UTC m=+448.565019722" watchObservedRunningTime="2026-07-24 13:17:09.342180426 +0000 UTC m=+448.566663731"
Jul 24 13:17:09.413217 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.413177    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-l6jqw"]
Jul 24 13:17:09.422218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.422187    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-l6jqw"]
Jul 24 13:17:09.538092 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.538062    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:09.639181 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.639087    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmm5\" (UniqueName: \"kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5\") pod \"f7345e5d-b124-4c99-a1bf-0f284473b105\" (UID: \"f7345e5d-b124-4c99-a1bf-0f284473b105\") "
Jul 24 13:17:09.641170 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.641132    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5" (OuterVolumeSpecName: "kube-api-access-bqmm5") pod "f7345e5d-b124-4c99-a1bf-0f284473b105" (UID: "f7345e5d-b124-4c99-a1bf-0f284473b105"). InnerVolumeSpecName "kube-api-access-bqmm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:17:09.740123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:09.740094    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmm5\" (UniqueName: \"kubernetes.io/projected/f7345e5d-b124-4c99-a1bf-0f284473b105-kube-api-access-bqmm5\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:17:10.318074 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.317998    2556 generic.go:356] "Generic (PLEG): container finished" podID="f7345e5d-b124-4c99-a1bf-0f284473b105" containerID="9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a" exitCode=0
Jul 24 13:17:10.318074 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.318039    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fxlxq"
Jul 24 13:17:10.318539 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.318095    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" event={"ID":"f7345e5d-b124-4c99-a1bf-0f284473b105","Type":"ContainerDied","Data":"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"}
Jul 24 13:17:10.318539 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.318138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fxlxq" event={"ID":"f7345e5d-b124-4c99-a1bf-0f284473b105","Type":"ContainerDied","Data":"a546278fca7e8bbaf33382d99f26b1bd718d85515a3a07d2c767f9f60eb8c1cc"}
Jul 24 13:17:10.318539 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.318163    2556 scope.go:117] "RemoveContainer" containerID="9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"
Jul 24 13:17:10.331733 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.331714    2556 scope.go:117] "RemoveContainer" containerID="9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"
Jul 24 13:17:10.332022 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:17:10.331989    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a\": container with ID starting with 9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a not found: ID does not exist" containerID="9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"
Jul 24 13:17:10.332076 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.332030    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a"} err="failed to get container status \"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a\": rpc error: code = NotFound desc = could not find container \"9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a\": container with ID starting with 9a6cd0cd3638caa5ee5b022b0efe6a4fd1b57e7806cd2f8c59963999d0f8ac7a not found: ID does not exist"
Jul 24 13:17:10.339956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.339915    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:10.343503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:10.343481    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fxlxq"]
Jul 24 13:17:11.341130 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:11.341101    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5767a2-0002-4431-9d87-1f7962ef27f0" path="/var/lib/kubelet/pods/db5767a2-0002-4431-9d87-1f7962ef27f0/volumes"
Jul 24 13:17:11.341494 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:17:11.341301    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7345e5d-b124-4c99-a1bf-0f284473b105" path="/var/lib/kubelet/pods/f7345e5d-b124-4c99-a1bf-0f284473b105/volumes"
Jul 24 13:18:12.623362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.623321    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"]
Jul 24 13:18:12.623957 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.623803    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7345e5d-b124-4c99-a1bf-0f284473b105" containerName="authorino"
Jul 24 13:18:12.623957 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.623821    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7345e5d-b124-4c99-a1bf-0f284473b105" containerName="authorino"
Jul 24 13:18:12.623957 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.623916    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7345e5d-b124-4c99-a1bf-0f284473b105" containerName="authorino"
Jul 24 13:18:12.627001 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.626982    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.630073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.630046    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 13:18:12.630073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.630064    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 13:18:12.630237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.630048    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4v65n\""
Jul 24 13:18:12.630237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.630056    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 24 13:18:12.637041 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.637019    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"]
Jul 24 13:18:12.792528 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.792528 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02d5035-d048-45de-8ea3-89bb0efd78dc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.792744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp648\" (UniqueName: \"kubernetes.io/projected/d02d5035-d048-45de-8ea3-89bb0efd78dc-kube-api-access-qp648\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.792744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.792744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.792744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.792693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02d5035-d048-45de-8ea3-89bb0efd78dc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893853 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp648\" (UniqueName: \"kubernetes.io/projected/d02d5035-d048-45de-8ea3-89bb0efd78dc-kube-api-access-qp648\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893853 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893966 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.893966 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.893893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.894088 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.894066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.894159 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.894140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.894257 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.894235    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.896162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.896141    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d02d5035-d048-45de-8ea3-89bb0efd78dc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.896304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.896288    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02d5035-d048-45de-8ea3-89bb0efd78dc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.902493 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.902471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp648\" (UniqueName: \"kubernetes.io/projected/d02d5035-d048-45de-8ea3-89bb0efd78dc-kube-api-access-qp648\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg\" (UID: \"d02d5035-d048-45de-8ea3-89bb0efd78dc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:12.938267 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:12.938235    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:13.073872 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:13.073851    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"]
Jul 24 13:18:13.076247 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:18:13.076217    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02d5035_d048_45de_8ea3_89bb0efd78dc.slice/crio-1974bf6dd23defd5984599d327ec86b10048a0cabbae7b32e1820b0fefa58bf4 WatchSource:0}: Error finding container 1974bf6dd23defd5984599d327ec86b10048a0cabbae7b32e1820b0fefa58bf4: Status 404 returned error can't find the container with id 1974bf6dd23defd5984599d327ec86b10048a0cabbae7b32e1820b0fefa58bf4
Jul 24 13:18:13.590901 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:13.590861    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg" event={"ID":"d02d5035-d048-45de-8ea3-89bb0efd78dc","Type":"ContainerStarted","Data":"1974bf6dd23defd5984599d327ec86b10048a0cabbae7b32e1820b0fefa58bf4"}
Jul 24 13:18:14.596813 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:14.596735    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg" event={"ID":"d02d5035-d048-45de-8ea3-89bb0efd78dc","Type":"ContainerStarted","Data":"5c8d2538c24cb0f5cdd840acf43c0fa3caf4b1cf69ad7e39d6fe686db0593204"}
Jul 24 13:18:14.614999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:14.614951    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg" podStartSLOduration=1.355008724 podStartE2EDuration="2.614916904s" totalImagesPullingTime="1.25990818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:18:14.614121114 +0000 UTC m=+513.838604442" watchObservedRunningTime="2026-07-24 13:18:14.614916904 +0000 UTC m=+513.839400231"
Jul 24 13:18:22.939288 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:22.939249    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:22.939700 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:22.939315    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:22.951728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:22.951698    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:23.646856 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:23.646827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg"
Jul 24 13:18:31.816831 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.816751    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"]
Jul 24 13:18:31.820404 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.820388    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.822562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.822543    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 13:18:31.827722 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.827703    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"]
Jul 24 13:18:31.862805 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.862774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.862994 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.862828    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.863051 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.862991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.863051 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.863017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.863051 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.863038    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.863162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.863056    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglws\" (UniqueName: \"kubernetes.io/projected/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-kube-api-access-mglws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963681    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963735    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963752    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglws\" (UniqueName: \"kubernetes.io/projected/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-kube-api-access-mglws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.963969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.963803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.964180 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.964156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.964180 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.964168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.964270 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.964246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.966226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.966193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.966446 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.966429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:31.971690 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:31.971673    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglws\" (UniqueName: \"kubernetes.io/projected/4ebfaf69-01f6-4719-87dd-1c99b76bdebc-kube-api-access-mglws\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wnhhc\" (UID: \"4ebfaf69-01f6-4719-87dd-1c99b76bdebc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:32.131288 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:32.131210    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:32.259606 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:32.259573    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"]
Jul 24 13:18:32.263551 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:18:32.263520    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebfaf69_01f6_4719_87dd_1c99b76bdebc.slice/crio-2e854069ea46f5e4a87d589fd8bfa4638ce0609ae0a8d4323041a4933c10b7c3 WatchSource:0}: Error finding container 2e854069ea46f5e4a87d589fd8bfa4638ce0609ae0a8d4323041a4933c10b7c3: Status 404 returned error can't find the container with id 2e854069ea46f5e4a87d589fd8bfa4638ce0609ae0a8d4323041a4933c10b7c3
Jul 24 13:18:32.676223 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:32.676186    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc" event={"ID":"4ebfaf69-01f6-4719-87dd-1c99b76bdebc","Type":"ContainerStarted","Data":"de8415ff429c6d028670cf850719c977a681c6d406eac2f16a68b523833e24cd"}
Jul 24 13:18:32.676411 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:32.676236    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc" event={"ID":"4ebfaf69-01f6-4719-87dd-1c99b76bdebc","Type":"ContainerStarted","Data":"2e854069ea46f5e4a87d589fd8bfa4638ce0609ae0a8d4323041a4933c10b7c3"}
Jul 24 13:18:32.692095 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:32.692047    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc" podStartSLOduration=1.451708631 podStartE2EDuration="1.692033231s" totalImagesPullingTime="240.3246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:18:32.690710711 +0000 UTC m=+531.915194030" watchObservedRunningTime="2026-07-24 13:18:32.692033231 +0000 UTC m=+531.916516557"
Jul 24 13:18:42.131464 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:42.131428    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:42.131904 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:42.131503    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:42.143760 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:42.143736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:18:42.732707 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:18:42.732678    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wnhhc"
Jul 24 13:19:15.403686 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.403650    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"]
Jul 24 13:19:15.409513 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.409480    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.411708 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.411684    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 13:19:15.414529 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.414505    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"]
Jul 24 13:19:15.441588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.441730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.441730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.441730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.441730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516107c4-ee0d-48d1-aea9-31e33080dd58-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.441915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.441755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6w96\" (UniqueName: \"kubernetes.io/projected/516107c4-ee0d-48d1-aea9-31e33080dd58-kube-api-access-v6w96\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542359    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516107c4-ee0d-48d1-aea9-31e33080dd58-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6w96\" (UniqueName: \"kubernetes.io/projected/516107c4-ee0d-48d1-aea9-31e33080dd58-kube-api-access-v6w96\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542554    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.542772 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.543002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.542971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.543310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.543010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.543310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.543060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.544841 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.544819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/516107c4-ee0d-48d1-aea9-31e33080dd58-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.544947 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.544916    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516107c4-ee0d-48d1-aea9-31e33080dd58-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.549956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.549918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6w96\" (UniqueName: \"kubernetes.io/projected/516107c4-ee0d-48d1-aea9-31e33080dd58-kube-api-access-v6w96\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xlklr\" (UID: \"516107c4-ee0d-48d1-aea9-31e33080dd58\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.719851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.719770    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:15.851611 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.851585    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"]
Jul 24 13:19:15.854419 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:19:15.854389    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod516107c4_ee0d_48d1_aea9_31e33080dd58.slice/crio-5b36ea230efbee8fd8ec75b48189fd265e2b8f1f87f9a63fe79b1de658e5e2ad WatchSource:0}: Error finding container 5b36ea230efbee8fd8ec75b48189fd265e2b8f1f87f9a63fe79b1de658e5e2ad: Status 404 returned error can't find the container with id 5b36ea230efbee8fd8ec75b48189fd265e2b8f1f87f9a63fe79b1de658e5e2ad
Jul 24 13:19:15.870728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:15.870700    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr" event={"ID":"516107c4-ee0d-48d1-aea9-31e33080dd58","Type":"ContainerStarted","Data":"5b36ea230efbee8fd8ec75b48189fd265e2b8f1f87f9a63fe79b1de658e5e2ad"}
Jul 24 13:19:16.877557 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:16.877525    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr" event={"ID":"516107c4-ee0d-48d1-aea9-31e33080dd58","Type":"ContainerStarted","Data":"cf375119805c760de50c6a9c7088097136e516a33667cb1fa1475167f4c3d069"}
Jul 24 13:19:16.900197 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:16.900150    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr" podStartSLOduration=1.701505485 podStartE2EDuration="1.900134872s" totalImagesPullingTime="198.629387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:19:16.898463726 +0000 UTC m=+576.122947054" watchObservedRunningTime="2026-07-24 13:19:16.900134872 +0000 UTC m=+576.124618226"
Jul 24 13:19:25.720012 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:25.719972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:25.720517 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:25.720054    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:25.732284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:25.732261    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:25.934380 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:25.934349    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xlklr"
Jul 24 13:19:41.166148 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:41.166121    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:19:41.175365 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:19:41.175346    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:20:24.757435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.757406    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-856b56fb59-s9wjl"]
Jul 24 13:20:24.760493 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.760476    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.767901 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.767881    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856b56fb59-s9wjl"]
Jul 24 13:20:24.860850 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.860804    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxc7v\" (UniqueName: \"kubernetes.io/projected/b15b813a-57cb-47b1-8b35-089b69569975-kube-api-access-jxc7v\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.861029 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.860922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b15b813a-57cb-47b1-8b35-089b69569975-tls-cert\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.962178 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.962141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxc7v\" (UniqueName: \"kubernetes.io/projected/b15b813a-57cb-47b1-8b35-089b69569975-kube-api-access-jxc7v\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.962362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.962235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b15b813a-57cb-47b1-8b35-089b69569975-tls-cert\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.964780 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.964751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b15b813a-57cb-47b1-8b35-089b69569975-tls-cert\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:24.969533 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:24.969511    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxc7v\" (UniqueName: \"kubernetes.io/projected/b15b813a-57cb-47b1-8b35-089b69569975-kube-api-access-jxc7v\") pod \"authorino-856b56fb59-s9wjl\" (UID: \"b15b813a-57cb-47b1-8b35-089b69569975\") " pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:25.070821 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:25.070795    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856b56fb59-s9wjl"
Jul 24 13:20:25.205176 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:25.205153    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856b56fb59-s9wjl"]
Jul 24 13:20:25.207802 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:20:25.207772    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb15b813a_57cb_47b1_8b35_089b69569975.slice/crio-5536a3db4ec1d8ca0291edfd6407039865f060fb9058cb4800ec2b382cd7d8a4 WatchSource:0}: Error finding container 5536a3db4ec1d8ca0291edfd6407039865f060fb9058cb4800ec2b382cd7d8a4: Status 404 returned error can't find the container with id 5536a3db4ec1d8ca0291edfd6407039865f060fb9058cb4800ec2b382cd7d8a4
Jul 24 13:20:26.197452 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.196574    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856b56fb59-s9wjl" event={"ID":"b15b813a-57cb-47b1-8b35-089b69569975","Type":"ContainerStarted","Data":"4637a6ce7d15d80e616a0fcf19cc93bf5ac1c5672975c10ebc00ec1d01e5dfe5"}
Jul 24 13:20:26.197452 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.196623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856b56fb59-s9wjl" event={"ID":"b15b813a-57cb-47b1-8b35-089b69569975","Type":"ContainerStarted","Data":"5536a3db4ec1d8ca0291edfd6407039865f060fb9058cb4800ec2b382cd7d8a4"}
Jul 24 13:20:26.223160 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.223101    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-856b56fb59-s9wjl" podStartSLOduration=1.716106089 podStartE2EDuration="2.223084627s" totalImagesPullingTime="506.978538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:20:26.221150704 +0000 UTC m=+645.445634035" watchObservedRunningTime="2026-07-24 13:20:26.223084627 +0000 UTC m=+645.447567953"
Jul 24 13:20:26.277890 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.277846    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:20:26.278093 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.278035    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" podUID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" containerName="authorino" containerID="cri-o://ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447" gracePeriod=30
Jul 24 13:20:26.557184 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.557152    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:20:26.680377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.680333    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28j6x\" (UniqueName: \"kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x\") pod \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") "
Jul 24 13:20:26.680570 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.680388    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert\") pod \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\" (UID: \"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6\") "
Jul 24 13:20:26.682654 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.682618    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x" (OuterVolumeSpecName: "kube-api-access-28j6x") pod "a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" (UID: "a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6"). InnerVolumeSpecName "kube-api-access-28j6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:20:26.691744 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.691711    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" (UID: "a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:20:26.781205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.781168    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-tls-cert\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:20:26.781205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:26.781202    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28j6x\" (UniqueName: \"kubernetes.io/projected/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6-kube-api-access-28j6x\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:20:27.203778 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.203683    2556 generic.go:356] "Generic (PLEG): container finished" podID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" containerID="ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447" exitCode=0
Jul 24 13:20:27.203778 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.203750    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fdbf4595d-v42bw"
Jul 24 13:20:27.204239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.203781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" event={"ID":"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6","Type":"ContainerDied","Data":"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"}
Jul 24 13:20:27.204239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.203829    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fdbf4595d-v42bw" event={"ID":"a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6","Type":"ContainerDied","Data":"c1ec723e9b801205a2ec578243be27ddcb3187669fdf4fd86575a9294252ea31"}
Jul 24 13:20:27.204239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.203846    2556 scope.go:117] "RemoveContainer" containerID="ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"
Jul 24 13:20:27.213200 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.213180    2556 scope.go:117] "RemoveContainer" containerID="ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"
Jul 24 13:20:27.213489 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:20:27.213470    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447\": container with ID starting with ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447 not found: ID does not exist" containerID="ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"
Jul 24 13:20:27.213565 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.213502    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447"} err="failed to get container status \"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447\": rpc error: code = NotFound desc = could not find container \"ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447\": container with ID starting with ff31def5ec17df3c85e0cebaaf5bd58b99b80f5922d2add5c8d4e99aac94f447 not found: ID does not exist"
Jul 24 13:20:27.224471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.224431    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:20:27.228784 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.228756    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fdbf4595d-v42bw"]
Jul 24 13:20:27.341693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:20:27.341663    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" path="/var/lib/kubelet/pods/a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6/volumes"
Jul 24 13:24:41.200924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:24:41.200842    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:24:41.207300 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:24:41.207274    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:29:41.229259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:29:41.229231    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:29:41.231692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:29:41.231669    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:31:23.620344 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.620310    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:31:23.620836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.620789    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" containerName="authorino"
Jul 24 13:31:23.620836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.620803    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" containerName="authorino"
Jul 24 13:31:23.620919 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.620910    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0aa0c6a-7b42-494a-8dd9-c20a1d3bf2f6" containerName="authorino"
Jul 24 13:31:23.623673 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.623658    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.625854 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.625835    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c82697db-kserve-self-signed-certs\""
Jul 24 13:31:23.633699 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.633677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:31:23.737390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737356    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.737575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.737575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.737575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737468    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.737575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8jnx\" (UniqueName: \"kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.737724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.737570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.838967 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.838905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839127 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.838983    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839127 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839127 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8jnx\" (UniqueName: \"kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839127 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839114    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839307 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839373 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.839548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.839524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.841384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.841361    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.841601 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.841585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.846712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.846688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8jnx\" (UniqueName: \"kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx\") pod \"e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") " pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:23.934098 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:23.934012    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:24.069269 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:24.069235    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:31:24.072230 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:31:24.072202    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod373813cb_39c2_4d38_aa6b_4c9937ee247d.slice/crio-5f8421d9685267038e9568513493ef52da791afe247e69c350fa439d9ad7f49a WatchSource:0}: Error finding container 5f8421d9685267038e9568513493ef52da791afe247e69c350fa439d9ad7f49a: Status 404 returned error can't find the container with id 5f8421d9685267038e9568513493ef52da791afe247e69c350fa439d9ad7f49a
Jul 24 13:31:24.232819 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:24.232736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" event={"ID":"373813cb-39c2-4d38-aa6b-4c9937ee247d","Type":"ContainerStarted","Data":"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"}
Jul 24 13:31:24.232819 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:24.232776    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" event={"ID":"373813cb-39c2-4d38-aa6b-4c9937ee247d","Type":"ContainerStarted","Data":"5f8421d9685267038e9568513493ef52da791afe247e69c350fa439d9ad7f49a"}
Jul 24 13:31:24.251953 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:24.251886    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" podStartSLOduration=1.251870191 podStartE2EDuration="1.251870191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:31:24.249697954 +0000 UTC m=+1303.474181284" watchObservedRunningTime="2026-07-24 13:31:24.251870191 +0000 UTC m=+1303.476353517"
Jul 24 13:31:33.935102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:33.935066    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:33.935504 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:33.935125    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:33.947294 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:33.947266    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:31:34.289563 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:31:34.289534    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:32:01.044600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:01.044563    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:32:16.064373 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.064325    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" podUID="373813cb-39c2-4d38-aa6b-4c9937ee247d" containerName="main" containerID="cri-o://ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87" gracePeriod=45
Jul 24 13:32:16.312912 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.312886    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:32:16.402177 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402080    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402177 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402138    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8jnx\" (UniqueName: \"kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402177 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402172    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402451 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402206    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402451 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402255    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402451 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402286    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache\") pod \"373813cb-39c2-4d38-aa6b-4c9937ee247d\" (UID: \"373813cb-39c2-4d38-aa6b-4c9937ee247d\") "
Jul 24 13:32:16.402451 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402385    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home" (OuterVolumeSpecName: "home") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:16.402692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402489    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:16.402739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402686    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache" (OuterVolumeSpecName: "model-cache") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:16.402739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402705    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.402739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.402721    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.404562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.404519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx" (OuterVolumeSpecName: "kube-api-access-x8jnx") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "kube-api-access-x8jnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:32:16.404849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.404821    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm" (OuterVolumeSpecName: "dshm") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:32:16.404914 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.404893    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "373813cb-39c2-4d38-aa6b-4c9937ee247d" (UID: "373813cb-39c2-4d38-aa6b-4c9937ee247d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:32:16.474118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.474085    2556 generic.go:356] "Generic (PLEG): container finished" podID="373813cb-39c2-4d38-aa6b-4c9937ee247d" containerID="ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87" exitCode=0
Jul 24 13:32:16.474279 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.474149    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"
Jul 24 13:32:16.474279 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.474151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" event={"ID":"373813cb-39c2-4d38-aa6b-4c9937ee247d","Type":"ContainerDied","Data":"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"}
Jul 24 13:32:16.474392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.474274    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg" event={"ID":"373813cb-39c2-4d38-aa6b-4c9937ee247d","Type":"ContainerDied","Data":"5f8421d9685267038e9568513493ef52da791afe247e69c350fa439d9ad7f49a"}
Jul 24 13:32:16.474392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.474298    2556 scope.go:117] "RemoveContainer" containerID="ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"
Jul 24 13:32:16.484216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.484195    2556 scope.go:117] "RemoveContainer" containerID="ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"
Jul 24 13:32:16.484478 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:32:16.484457    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87\": container with ID starting with ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87 not found: ID does not exist" containerID="ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"
Jul 24 13:32:16.484546 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.484491    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87"} err="failed to get container status \"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87\": rpc error: code = NotFound desc = could not find container \"ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87\": container with ID starting with ecdee073e5a7ef5037503f3e30814e474d1b95ddb5c09cca83b0efa0b025bb87 not found: ID does not exist"
Jul 24 13:32:16.504144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.504114    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8jnx\" (UniqueName: \"kubernetes.io/projected/373813cb-39c2-4d38-aa6b-4c9937ee247d-kube-api-access-x8jnx\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.504144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.504145    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373813cb-39c2-4d38-aa6b-4c9937ee247d-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.504331 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.504162    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.504331 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.504177    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/373813cb-39c2-4d38-aa6b-4c9937ee247d-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:32:16.511407 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.511379    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:32:16.525450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:16.525423    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c82697db-kserve-7cbf5c8795-vnszg"]
Jul 24 13:32:17.342765 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:32:17.342723    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373813cb-39c2-4d38-aa6b-4c9937ee247d" path="/var/lib/kubelet/pods/373813cb-39c2-4d38-aa6b-4c9937ee247d/volumes"
Jul 24 13:34:41.256752 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:34:41.256715    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:34:41.257356 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:34:41.256839    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:35:48.866244 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.866207    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:35:48.868387 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.866570    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373813cb-39c2-4d38-aa6b-4c9937ee247d" containerName="main"
Jul 24 13:35:48.868387 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.866582    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="373813cb-39c2-4d38-aa6b-4c9937ee247d" containerName="main"
Jul 24 13:35:48.868387 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.866660    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="373813cb-39c2-4d38-aa6b-4c9937ee247d" containerName="main"
Jul 24 13:35:48.869735 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.869720    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:48.872286 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.872260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-77101517\""
Jul 24 13:35:48.872394 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.872262    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-77101517-dockercfg-bqhw8\""
Jul 24 13:35:48.880516 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.880496    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:35:48.905073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.905045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8hz\" (UniqueName: \"kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:48.905185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:48.905166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.006589 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.006522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.006759 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.006626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8hz\" (UniqueName: \"kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.007250 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.007231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.015021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.014994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8hz\" (UniqueName: \"kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz\") pod \"payload-processing-e2e-ait-77101517-788b584b94-gwmbf\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") " pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.180810 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.180713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:49.311803 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.311780    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:35:49.314457 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:35:49.314431    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf44d7b1_3a89_4dd3_a9a7_43a4948adb42.slice/crio-cd1ae70fbeaac0f4524941a5ca00df4df18dcdf9a5cc208705bf07a0d418897f WatchSource:0}: Error finding container cd1ae70fbeaac0f4524941a5ca00df4df18dcdf9a5cc208705bf07a0d418897f: Status 404 returned error can't find the container with id cd1ae70fbeaac0f4524941a5ca00df4df18dcdf9a5cc208705bf07a0d418897f
Jul 24 13:35:49.316401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.316384    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:35:49.476858 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:49.476781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" event={"ID":"df44d7b1-3a89-4dd3-a9a7-43a4948adb42","Type":"ContainerStarted","Data":"cd1ae70fbeaac0f4524941a5ca00df4df18dcdf9a5cc208705bf07a0d418897f"}
Jul 24 13:35:51.487712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:51.487670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" event={"ID":"df44d7b1-3a89-4dd3-a9a7-43a4948adb42","Type":"ContainerStarted","Data":"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"}
Jul 24 13:35:51.488122 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:51.487809    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:35:51.504257 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:35:51.504208    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" podStartSLOduration=2.129209955 podStartE2EDuration="3.504193825s" totalImagesPullingTime="1.37498387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:51.503280798 +0000 UTC m=+1570.727764137" watchObservedRunningTime="2026-07-24 13:35:51.504193825 +0000 UTC m=+1570.728677152"
Jul 24 13:36:02.494438 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:02.494406    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:36:13.558857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.558815    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"]
Jul 24 13:36:13.562059 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.562044    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:13.564352 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.564316    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:36:13.568459 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.568437    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"]
Jul 24 13:36:13.731055 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.731017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77101517-j7d2n\" (UID: \"cd77303b-b98d-41aa-87be-bbe5a08db4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:13.831864 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.831791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77101517-j7d2n\" (UID: \"cd77303b-b98d-41aa-87be-bbe5a08db4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:13.832414 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.832391    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77101517-j7d2n\" (UID: \"cd77303b-b98d-41aa-87be-bbe5a08db4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:13.873838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:13.873807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:14.006758 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:14.006726    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"]
Jul 24 13:36:14.009389 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:36:14.009363    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd77303b_b98d_41aa_87be_bbe5a08db4ce.slice/crio-cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2 WatchSource:0}: Error finding container cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2: Status 404 returned error can't find the container with id cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2
Jul 24 13:36:14.596780 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:14.596746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n" event={"ID":"cd77303b-b98d-41aa-87be-bbe5a08db4ce","Type":"ContainerStarted","Data":"cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2"}
Jul 24 13:36:16.609192 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:16.609159    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n" event={"ID":"cd77303b-b98d-41aa-87be-bbe5a08db4ce","Type":"ContainerStarted","Data":"cdb7d32c1836717742ddc2a54f8b2ee267a990af7722abbf9b439709012b547b"}
Jul 24 13:36:16.625807 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:16.625750    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n" podStartSLOduration=1.5338697730000002 podStartE2EDuration="3.62573478s" totalImagesPullingTime="2.091865007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:16.624702637 +0000 UTC m=+1595.849185964" watchObservedRunningTime="2026-07-24 13:36:16.62573478 +0000 UTC m=+1595.850218106"
Jul 24 13:36:17.615240 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:17.615201    2556 generic.go:356] "Generic (PLEG): container finished" podID="cd77303b-b98d-41aa-87be-bbe5a08db4ce" containerID="cdb7d32c1836717742ddc2a54f8b2ee267a990af7722abbf9b439709012b547b" exitCode=0
Jul 24 13:36:17.615620 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:17.615285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n" event={"ID":"cd77303b-b98d-41aa-87be-bbe5a08db4ce","Type":"ContainerDied","Data":"cdb7d32c1836717742ddc2a54f8b2ee267a990af7722abbf9b439709012b547b"}
Jul 24 13:36:18.748955 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:18.748907    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:18.774924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:18.774899    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca\") pod \"cd77303b-b98d-41aa-87be-bbe5a08db4ce\" (UID: \"cd77303b-b98d-41aa-87be-bbe5a08db4ce\") "
Jul 24 13:36:18.775209 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:18.775185    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd77303b-b98d-41aa-87be-bbe5a08db4ce" (UID: "cd77303b-b98d-41aa-87be-bbe5a08db4ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:18.875612 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:18.875531    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77303b-b98d-41aa-87be-bbe5a08db4ce-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:36:19.625299 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:19.625221    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"
Jul 24 13:36:19.625299 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:19.625264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n" event={"ID":"cd77303b-b98d-41aa-87be-bbe5a08db4ce","Type":"ContainerDied","Data":"cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2"}
Jul 24 13:36:19.625512 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:19.625307    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc89e74c685b8326191e4b1958473d103b241c3d0e0417f82a5c91e7fc2395e2"
Jul 24 13:36:24.063589 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.063556    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:36:24.064009 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.063773    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" podUID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" containerName="payload-processing" containerID="cri-o://c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b" gracePeriod=30
Jul 24 13:36:24.311108 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.311080    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:36:24.325308 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.325257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume\") pod \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") "
Jul 24 13:36:24.325415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.325402    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8hz\" (UniqueName: \"kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz\") pod \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\" (UID: \"df44d7b1-3a89-4dd3-a9a7-43a4948adb42\") "
Jul 24 13:36:24.325653 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.325627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df44d7b1-3a89-4dd3-a9a7-43a4948adb42" (UID: "df44d7b1-3a89-4dd3-a9a7-43a4948adb42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:24.328426 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.328386    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz" (OuterVolumeSpecName: "kube-api-access-5t8hz") pod "df44d7b1-3a89-4dd3-a9a7-43a4948adb42" (UID: "df44d7b1-3a89-4dd3-a9a7-43a4948adb42"). InnerVolumeSpecName "kube-api-access-5t8hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:36:24.426581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.426551    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8hz\" (UniqueName: \"kubernetes.io/projected/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-kube-api-access-5t8hz\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:36:24.426581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.426575    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d7b1-3a89-4dd3-a9a7-43a4948adb42-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:36:24.656352 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.656267    2556 generic.go:356] "Generic (PLEG): container finished" podID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" containerID="c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b" exitCode=0
Jul 24 13:36:24.656352 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.656310    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"
Jul 24 13:36:24.656526 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.656363    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" event={"ID":"df44d7b1-3a89-4dd3-a9a7-43a4948adb42","Type":"ContainerDied","Data":"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"}
Jul 24 13:36:24.656526 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.656418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf" event={"ID":"df44d7b1-3a89-4dd3-a9a7-43a4948adb42","Type":"ContainerDied","Data":"cd1ae70fbeaac0f4524941a5ca00df4df18dcdf9a5cc208705bf07a0d418897f"}
Jul 24 13:36:24.656526 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.656446    2556 scope.go:117] "RemoveContainer" containerID="c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"
Jul 24 13:36:24.665244 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.665227    2556 scope.go:117] "RemoveContainer" containerID="c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"
Jul 24 13:36:24.665493 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:36:24.665475    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b\": container with ID starting with c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b not found: ID does not exist" containerID="c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"
Jul 24 13:36:24.665549 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.665501    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b"} err="failed to get container status \"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b\": rpc error: code = NotFound desc = could not find container \"c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b\": container with ID starting with c6fdc4c060bd6442f7bf5c46ddd1483474c3213b83789f305182a52b5c3d3a5b not found: ID does not exist"
Jul 24 13:36:24.683813 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.683790    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:36:24.690023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:24.690003    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77101517-788b584b94-gwmbf"]
Jul 24 13:36:25.340696 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:25.340664    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" path="/var/lib/kubelet/pods/df44d7b1-3a89-4dd3-a9a7-43a4948adb42/volumes"
Jul 24 13:36:28.637006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:28.636971    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"]
Jul 24 13:36:28.640152 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:28.640129    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77101517-j7d2n"]
Jul 24 13:36:29.340989 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:29.340923    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd77303b-b98d-41aa-87be-bbe5a08db4ce" path="/var/lib/kubelet/pods/cd77303b-b98d-41aa-87be-bbe5a08db4ce/volumes"
Jul 24 13:36:32.380620 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.380535    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.380915    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd77303b-b98d-41aa-87be-bbe5a08db4ce" containerName="revoke-keys"
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.380948    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd77303b-b98d-41aa-87be-bbe5a08db4ce" containerName="revoke-keys"
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.380970    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" containerName="payload-processing"
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.380976    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" containerName="payload-processing"
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.381040    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="df44d7b1-3a89-4dd3-a9a7-43a4948adb42" containerName="payload-processing"
Jul 24 13:36:32.381056 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.381049    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd77303b-b98d-41aa-87be-bbe5a08db4ce" containerName="revoke-keys"
Jul 24 13:36:32.385540 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.385525    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.388208 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.388185    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:36:32.388441 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.388426    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-51268fa8-serving-cert\""
Jul 24 13:36:32.401093 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.401071    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:36:32.498802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.498766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.498802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.498804    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jw9\" (UniqueName: \"kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.499036 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.498827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.599434 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.599401    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.599553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.599438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jw9\" (UniqueName: \"kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.599553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.599462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.599625 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:36:32.599596    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-serving-cert: secret "maas-api-e2e-migrate-51268fa8-serving-cert" not found
Jul 24 13:36:32.599668 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:36:32.599659    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls podName:397f45d0-8114-4754-9eec-3ea3d169f296 nodeName:}" failed. No retries permitted until 2026-07-24 13:36:33.099642722 +0000 UTC m=+1612.324126028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls") pod "maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" (UID: "397f45d0-8114-4754-9eec-3ea3d169f296") : secret "maas-api-e2e-migrate-51268fa8-serving-cert" not found
Jul 24 13:36:32.600115 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.600097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:32.610263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:32.610244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jw9\" (UniqueName: \"kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:33.104051 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.104001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:33.106627 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.106587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") pod \"maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:33.190422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.190382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:36:33.193844 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.193828    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.196139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.196116    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-51268fa8-dockercfg-nl2fc\""
Jul 24 13:36:33.196275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.196228    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-51268fa8\""
Jul 24 13:36:33.202370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.202348    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:36:33.295339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.295295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:33.305593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.305565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.305704 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.305624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxtf\" (UniqueName: \"kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.406399 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.406367    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxtf\" (UniqueName: \"kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.406752 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.406450    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.407121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.407101    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.414762 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.414734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxtf\" (UniqueName: \"kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf\") pod \"payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.428377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.428356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:36:33.431028 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:36:33.431005    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397f45d0_8114_4754_9eec_3ea3d169f296.slice/crio-b8a28a3317cbb91d6b3f07ba5f025609edfaf3419554da791c3eb97547c26599 WatchSource:0}: Error finding container b8a28a3317cbb91d6b3f07ba5f025609edfaf3419554da791c3eb97547c26599: Status 404 returned error can't find the container with id b8a28a3317cbb91d6b3f07ba5f025609edfaf3419554da791c3eb97547c26599
Jul 24 13:36:33.505255 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.505221    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:33.637919 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.637895    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:36:33.640139 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:36:33.640100    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3620eb7_0e28_4c9b_9164_fcf65aad60d0.slice/crio-5f9a3b55031bbf5551b8ed545c75af9ccb87bf619321abef4253b4ee741ccecd WatchSource:0}: Error finding container 5f9a3b55031bbf5551b8ed545c75af9ccb87bf619321abef4253b4ee741ccecd: Status 404 returned error can't find the container with id 5f9a3b55031bbf5551b8ed545c75af9ccb87bf619321abef4253b4ee741ccecd
Jul 24 13:36:33.699655 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.699628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" event={"ID":"397f45d0-8114-4754-9eec-3ea3d169f296","Type":"ContainerStarted","Data":"b8a28a3317cbb91d6b3f07ba5f025609edfaf3419554da791c3eb97547c26599"}
Jul 24 13:36:33.701339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:33.701306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" event={"ID":"e3620eb7-0e28-4c9b-9164-fcf65aad60d0","Type":"ContainerStarted","Data":"5f9a3b55031bbf5551b8ed545c75af9ccb87bf619321abef4253b4ee741ccecd"}
Jul 24 13:36:34.708107 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:34.708074    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" event={"ID":"e3620eb7-0e28-4c9b-9164-fcf65aad60d0","Type":"ContainerStarted","Data":"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"}
Jul 24 13:36:34.708530 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:34.708212    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:34.726368 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:34.726304    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" podStartSLOduration=1.726287044 podStartE2EDuration="1.726287044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:34.725359655 +0000 UTC m=+1613.949842990" watchObservedRunningTime="2026-07-24 13:36:34.726287044 +0000 UTC m=+1613.950770370"
Jul 24 13:36:36.722579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:36.722541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" event={"ID":"397f45d0-8114-4754-9eec-3ea3d169f296","Type":"ContainerStarted","Data":"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"}
Jul 24 13:36:36.722969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:36.722617    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:36.741164 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:36.741114    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" podStartSLOduration=2.47807245 podStartE2EDuration="4.741098264s" totalImagesPullingTime="2.263025814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:36.739213794 +0000 UTC m=+1615.963697115" watchObservedRunningTime="2026-07-24 13:36:36.741098264 +0000 UTC m=+1615.965581643"
Jul 24 13:36:42.733770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:42.733742    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:36:45.718304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:45.718274    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:36:57.262173 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.262132    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"]
Jul 24 13:36:57.269793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.269775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:36:57.272626 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.272601    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:36:57.274392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.274369    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"]
Jul 24 13:36:57.426750 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.426713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk\" (UID: \"7437d791-860e-44e1-a273-ad2a4e4dc599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:36:57.528149 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.528117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk\" (UID: \"7437d791-860e-44e1-a273-ad2a4e4dc599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:36:57.528786 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.528765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk\" (UID: \"7437d791-860e-44e1-a273-ad2a4e4dc599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:36:57.583102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.583074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:36:57.713067 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.713045    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"]
Jul 24 13:36:57.715033 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:36:57.715009    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7437d791_860e_44e1_a273_ad2a4e4dc599.slice/crio-b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e WatchSource:0}: Error finding container b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e: Status 404 returned error can't find the container with id b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e
Jul 24 13:36:57.825064 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:57.824981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk" event={"ID":"7437d791-860e-44e1-a273-ad2a4e4dc599","Type":"ContainerStarted","Data":"b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e"}
Jul 24 13:36:58.831303 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:58.831268    2556 generic.go:356] "Generic (PLEG): container finished" podID="7437d791-860e-44e1-a273-ad2a4e4dc599" containerID="980dacd6bf876a468bd7ab7828f35290e235f7f6d32e0aa25f34e9c036370df7" exitCode=0
Jul 24 13:36:58.831677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:58.831356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk" event={"ID":"7437d791-860e-44e1-a273-ad2a4e4dc599","Type":"ContainerDied","Data":"980dacd6bf876a468bd7ab7828f35290e235f7f6d32e0aa25f34e9c036370df7"}
Jul 24 13:36:59.961464 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:36:59.961430    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:37:00.047819 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.047782    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca\") pod \"7437d791-860e-44e1-a273-ad2a4e4dc599\" (UID: \"7437d791-860e-44e1-a273-ad2a4e4dc599\") "
Jul 24 13:37:00.048196 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.048177    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7437d791-860e-44e1-a273-ad2a4e4dc599" (UID: "7437d791-860e-44e1-a273-ad2a4e4dc599"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:00.149237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.149155    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7437d791-860e-44e1-a273-ad2a4e4dc599-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:00.841089 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.841056    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk" event={"ID":"7437d791-860e-44e1-a273-ad2a4e4dc599","Type":"ContainerDied","Data":"b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e"}
Jul 24 13:37:00.841263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.841141    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b16d5fc18c4aead18197d4b7ffe462e0a0c9db33d46038dcea6173f77c85230e"
Jul 24 13:37:00.841263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:00.841152    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"
Jul 24 13:37:07.853497 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:07.853466    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:37:07.854010 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:07.853659    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" podUID="397f45d0-8114-4754-9eec-3ea3d169f296" containerName="maas-api" containerID="cri-o://2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065" gracePeriod=30
Jul 24 13:37:07.913833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:07.913802    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:37:07.914063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:07.914039    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" podUID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" containerName="payload-pre-processing" containerID="cri-o://7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df" gracePeriod=30
Jul 24 13:37:08.153632 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.153609    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:37:08.182871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.182847    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:37:08.226396 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226365    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzxtf\" (UniqueName: \"kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf\") pod \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") "
Jul 24 13:37:08.226554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226437    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume\") pod \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\" (UID: \"e3620eb7-0e28-4c9b-9164-fcf65aad60d0\") "
Jul 24 13:37:08.226554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226484    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24jw9\" (UniqueName: \"kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9\") pod \"397f45d0-8114-4754-9eec-3ea3d169f296\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") "
Jul 24 13:37:08.226554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca\") pod \"397f45d0-8114-4754-9eec-3ea3d169f296\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") "
Jul 24 13:37:08.226554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226528    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") pod \"397f45d0-8114-4754-9eec-3ea3d169f296\" (UID: \"397f45d0-8114-4754-9eec-3ea3d169f296\") "
Jul 24 13:37:08.226885 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3620eb7-0e28-4c9b-9164-fcf65aad60d0" (UID: "e3620eb7-0e28-4c9b-9164-fcf65aad60d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:08.226885 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.226864    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "397f45d0-8114-4754-9eec-3ea3d169f296" (UID: "397f45d0-8114-4754-9eec-3ea3d169f296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:08.228631 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.228605    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf" (OuterVolumeSpecName: "kube-api-access-vzxtf") pod "e3620eb7-0e28-4c9b-9164-fcf65aad60d0" (UID: "e3620eb7-0e28-4c9b-9164-fcf65aad60d0"). InnerVolumeSpecName "kube-api-access-vzxtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:08.228751 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.228728    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "397f45d0-8114-4754-9eec-3ea3d169f296" (UID: "397f45d0-8114-4754-9eec-3ea3d169f296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:37:08.228792 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.228729    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9" (OuterVolumeSpecName: "kube-api-access-24jw9") pod "397f45d0-8114-4754-9eec-3ea3d169f296" (UID: "397f45d0-8114-4754-9eec-3ea3d169f296"). InnerVolumeSpecName "kube-api-access-24jw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:08.327386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.327344    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jw9\" (UniqueName: \"kubernetes.io/projected/397f45d0-8114-4754-9eec-3ea3d169f296-kube-api-access-24jw9\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:08.327386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.327379    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/397f45d0-8114-4754-9eec-3ea3d169f296-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:08.327386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.327390    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/397f45d0-8114-4754-9eec-3ea3d169f296-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:08.327386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.327398    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxtf\" (UniqueName: \"kubernetes.io/projected/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-kube-api-access-vzxtf\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:08.327694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.327407    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3620eb7-0e28-4c9b-9164-fcf65aad60d0-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:08.879486 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.879453    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" containerID="7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df" exitCode=0
Jul 24 13:37:08.879862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.879501    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"
Jul 24 13:37:08.879862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.879543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" event={"ID":"e3620eb7-0e28-4c9b-9164-fcf65aad60d0","Type":"ContainerDied","Data":"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"}
Jul 24 13:37:08.879862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.879595    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7" event={"ID":"e3620eb7-0e28-4c9b-9164-fcf65aad60d0","Type":"ContainerDied","Data":"5f9a3b55031bbf5551b8ed545c75af9ccb87bf619321abef4253b4ee741ccecd"}
Jul 24 13:37:08.879862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.879624    2556 scope.go:117] "RemoveContainer" containerID="7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"
Jul 24 13:37:08.880871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.880837    2556 generic.go:356] "Generic (PLEG): container finished" podID="397f45d0-8114-4754-9eec-3ea3d169f296" containerID="2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065" exitCode=0
Jul 24 13:37:08.880980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.880890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" event={"ID":"397f45d0-8114-4754-9eec-3ea3d169f296","Type":"ContainerDied","Data":"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"}
Jul 24 13:37:08.880980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.880901    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"
Jul 24 13:37:08.880980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.880957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc" event={"ID":"397f45d0-8114-4754-9eec-3ea3d169f296","Type":"ContainerDied","Data":"b8a28a3317cbb91d6b3f07ba5f025609edfaf3419554da791c3eb97547c26599"}
Jul 24 13:37:08.889077 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.889060    2556 scope.go:117] "RemoveContainer" containerID="7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"
Jul 24 13:37:08.889323 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:37:08.889306    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df\": container with ID starting with 7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df not found: ID does not exist" containerID="7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"
Jul 24 13:37:08.889392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.889330    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df"} err="failed to get container status \"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df\": rpc error: code = NotFound desc = could not find container \"7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df\": container with ID starting with 7030d37e6596f4ff130a1f797fecd7973223e7902ec60cdfab2ebbce798c28df not found: ID does not exist"
Jul 24 13:37:08.889392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.889345    2556 scope.go:117] "RemoveContainer" containerID="2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"
Jul 24 13:37:08.897267 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.897247    2556 scope.go:117] "RemoveContainer" containerID="2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"
Jul 24 13:37:08.897519 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:37:08.897498    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065\": container with ID starting with 2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065 not found: ID does not exist" containerID="2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"
Jul 24 13:37:08.897612 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.897524    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065"} err="failed to get container status \"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065\": rpc error: code = NotFound desc = could not find container \"2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065\": container with ID starting with 2ec8b4830e800a5eac5ea78c7ceaa59f2dc717d30a265759a705dec61ec4f065 not found: ID does not exist"
Jul 24 13:37:08.900028 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.899995    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:37:08.903843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.903820    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-51268fa8-945d996b4-xmhs7"]
Jul 24 13:37:08.913725 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.913706    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:37:08.919845 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:08.919826    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-51268fa8-59649fbf7d-g84nc"]
Jul 24 13:37:09.341304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:09.341274    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="397f45d0-8114-4754-9eec-3ea3d169f296" path="/var/lib/kubelet/pods/397f45d0-8114-4754-9eec-3ea3d169f296/volumes"
Jul 24 13:37:09.341615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:09.341602    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" path="/var/lib/kubelet/pods/e3620eb7-0e28-4c9b-9164-fcf65aad60d0/volumes"
Jul 24 13:37:12.401041 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:12.401004    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"]
Jul 24 13:37:12.403151 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:12.403129    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-51268fa8-h47hk"]
Jul 24 13:37:13.340886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:13.340851    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7437d791-860e-44e1-a273-ad2a4e4dc599" path="/var/lib/kubelet/pods/7437d791-860e-44e1-a273-ad2a4e4dc599/volumes"
Jul 24 13:37:15.417062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417030    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417356    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7437d791-860e-44e1-a273-ad2a4e4dc599" containerName="revoke-keys"
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417367    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7437d791-860e-44e1-a273-ad2a4e4dc599" containerName="revoke-keys"
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417382    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397f45d0-8114-4754-9eec-3ea3d169f296" containerName="maas-api"
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417387    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="397f45d0-8114-4754-9eec-3ea3d169f296" containerName="maas-api"
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417400    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" containerName="payload-pre-processing"
Jul 24 13:37:15.417413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417405    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" containerName="payload-pre-processing"
Jul 24 13:37:15.417597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417459    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3620eb7-0e28-4c9b-9164-fcf65aad60d0" containerName="payload-pre-processing"
Jul 24 13:37:15.417597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417469    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="397f45d0-8114-4754-9eec-3ea3d169f296" containerName="maas-api"
Jul 24 13:37:15.417597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.417476    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7437d791-860e-44e1-a273-ad2a4e4dc599" containerName="revoke-keys"
Jul 24 13:37:15.419618 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.419603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.422093 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.422066    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca810d9a-serving-cert\""
Jul 24 13:37:15.422459 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.422445    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:37:15.430102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.430065    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:15.492604 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.492574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.492732 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.492641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.492732 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.492662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvp9\" (UniqueName: \"kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.593321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.593286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.593321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.593322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvp9\" (UniqueName: \"kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.593520 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.593391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.593567 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:37:15.593544    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-serving-cert: secret "maas-api-e2e-ait-ca810d9a-serving-cert" not found
Jul 24 13:37:15.593659 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:37:15.593647    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls podName:c1fe53a1-4912-4f12-9a3a-2f53aba589e4 nodeName:}" failed. No retries permitted until 2026-07-24 13:37:16.093623629 +0000 UTC m=+1655.318106934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls") pod "maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" (UID: "c1fe53a1-4912-4f12-9a3a-2f53aba589e4") : secret "maas-api-e2e-ait-ca810d9a-serving-cert" not found
Jul 24 13:37:15.594039 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.594023    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:15.604305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:15.604285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvp9\" (UniqueName: \"kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:16.097495 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:16.097460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:16.099909 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:16.099888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") pod \"maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:16.330183 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:16.330144    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:16.497634 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:16.497610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:16.499143 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:37:16.499117    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fe53a1_4912_4f12_9a3a_2f53aba589e4.slice/crio-f0740be537bc857ad52beb54ae408bcaea7fe5410bb55a4dee07e8647b6b6c67 WatchSource:0}: Error finding container f0740be537bc857ad52beb54ae408bcaea7fe5410bb55a4dee07e8647b6b6c67: Status 404 returned error can't find the container with id f0740be537bc857ad52beb54ae408bcaea7fe5410bb55a4dee07e8647b6b6c67
Jul 24 13:37:16.927639 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:16.927611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" event={"ID":"c1fe53a1-4912-4f12-9a3a-2f53aba589e4","Type":"ContainerStarted","Data":"f0740be537bc857ad52beb54ae408bcaea7fe5410bb55a4dee07e8647b6b6c67"}
Jul 24 13:37:17.933780 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:17.933745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" event={"ID":"c1fe53a1-4912-4f12-9a3a-2f53aba589e4","Type":"ContainerStarted","Data":"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"}
Jul 24 13:37:17.934275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:17.933904    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:17.952167 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:17.952117    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" podStartSLOduration=2.54814214 podStartE2EDuration="2.952101629s" totalImagesPullingTime="403.959489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:17.950954809 +0000 UTC m=+1657.175438130" watchObservedRunningTime="2026-07-24 13:37:17.952101629 +0000 UTC m=+1657.176584955"
Jul 24 13:37:23.944605 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:23.944575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:41.239986 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.239945    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"]
Jul 24 13:37:41.243354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.243333    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:41.245557 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.245538    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:37:41.250678 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.250566    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"]
Jul 24 13:37:41.333431 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.333394    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg\" (UID: \"96b0b353-7a85-4bdb-a259-f3214356dcfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:41.434384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.434347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg\" (UID: \"96b0b353-7a85-4bdb-a259-f3214356dcfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:41.435174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.435148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg\" (UID: \"96b0b353-7a85-4bdb-a259-f3214356dcfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:41.554532 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.554502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:41.693158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:41.693127    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"]
Jul 24 13:37:41.696329 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:37:41.696304    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b0b353_7a85_4bdb_a259_f3214356dcfd.slice/crio-e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8 WatchSource:0}: Error finding container e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8: Status 404 returned error can't find the container with id e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8
Jul 24 13:37:42.046891 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:42.046866    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg" event={"ID":"96b0b353-7a85-4bdb-a259-f3214356dcfd","Type":"ContainerStarted","Data":"0c51346d69f05a50a5a1420ce2804db9d1bf70baa43fc5f7cbdfce4ad5837240"}
Jul 24 13:37:42.047022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:42.046909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg" event={"ID":"96b0b353-7a85-4bdb-a259-f3214356dcfd","Type":"ContainerStarted","Data":"e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8"}
Jul 24 13:37:42.060218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:42.060177    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg" podStartSLOduration=1.060163198 podStartE2EDuration="1.060163198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:37:42.059456369 +0000 UTC m=+1681.283939693" watchObservedRunningTime="2026-07-24 13:37:42.060163198 +0000 UTC m=+1681.284646569"
Jul 24 13:37:43.053409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:43.053361    2556 generic.go:356] "Generic (PLEG): container finished" podID="96b0b353-7a85-4bdb-a259-f3214356dcfd" containerID="0c51346d69f05a50a5a1420ce2804db9d1bf70baa43fc5f7cbdfce4ad5837240" exitCode=0
Jul 24 13:37:43.053764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:43.053450    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg" event={"ID":"96b0b353-7a85-4bdb-a259-f3214356dcfd","Type":"ContainerDied","Data":"0c51346d69f05a50a5a1420ce2804db9d1bf70baa43fc5f7cbdfce4ad5837240"}
Jul 24 13:37:44.185410 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:44.185388    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:44.257918 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:44.257888    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca\") pod \"96b0b353-7a85-4bdb-a259-f3214356dcfd\" (UID: \"96b0b353-7a85-4bdb-a259-f3214356dcfd\") "
Jul 24 13:37:44.258239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:44.258216    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96b0b353-7a85-4bdb-a259-f3214356dcfd" (UID: "96b0b353-7a85-4bdb-a259-f3214356dcfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:44.358955 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:44.358856    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96b0b353-7a85-4bdb-a259-f3214356dcfd-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:45.064577 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:45.064546    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"
Jul 24 13:37:45.064770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:45.064545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg" event={"ID":"96b0b353-7a85-4bdb-a259-f3214356dcfd","Type":"ContainerDied","Data":"e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8"}
Jul 24 13:37:45.064770 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:45.064688    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2898e75b918e9bdbd0296884c36bd3ec3ef73a777e78cdf96d47eaf6027eec8"
Jul 24 13:37:51.699747 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:51.699713    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:51.700205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:51.699948    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" podUID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" containerName="maas-api" containerID="cri-o://aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5" gracePeriod=30
Jul 24 13:37:51.939699 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:51.939672    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:52.027463 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.027433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dvp9\" (UniqueName: \"kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9\") pod \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") "
Jul 24 13:37:52.027641 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.027554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") pod \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") "
Jul 24 13:37:52.027641 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.027605    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca\") pod \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\" (UID: \"c1fe53a1-4912-4f12-9a3a-2f53aba589e4\") "
Jul 24 13:37:52.028020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.027982    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1fe53a1-4912-4f12-9a3a-2f53aba589e4" (UID: "c1fe53a1-4912-4f12-9a3a-2f53aba589e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:52.029619 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.029595    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9" (OuterVolumeSpecName: "kube-api-access-4dvp9") pod "c1fe53a1-4912-4f12-9a3a-2f53aba589e4" (UID: "c1fe53a1-4912-4f12-9a3a-2f53aba589e4"). InnerVolumeSpecName "kube-api-access-4dvp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:52.029727 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.029663    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1fe53a1-4912-4f12-9a3a-2f53aba589e4" (UID: "c1fe53a1-4912-4f12-9a3a-2f53aba589e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:37:52.099238 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.099204    2556 generic.go:356] "Generic (PLEG): container finished" podID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" containerID="aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5" exitCode=0
Jul 24 13:37:52.099409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.099247    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"
Jul 24 13:37:52.099409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.099282    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" event={"ID":"c1fe53a1-4912-4f12-9a3a-2f53aba589e4","Type":"ContainerDied","Data":"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"}
Jul 24 13:37:52.099409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.099339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc" event={"ID":"c1fe53a1-4912-4f12-9a3a-2f53aba589e4","Type":"ContainerDied","Data":"f0740be537bc857ad52beb54ae408bcaea7fe5410bb55a4dee07e8647b6b6c67"}
Jul 24 13:37:52.099409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.099369    2556 scope.go:117] "RemoveContainer" containerID="aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"
Jul 24 13:37:52.108738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.108720    2556 scope.go:117] "RemoveContainer" containerID="aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"
Jul 24 13:37:52.109007 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:37:52.108987    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5\": container with ID starting with aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5 not found: ID does not exist" containerID="aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"
Jul 24 13:37:52.109061 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.109015    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5"} err="failed to get container status \"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5\": rpc error: code = NotFound desc = could not find container \"aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5\": container with ID starting with aeb2a2852ef5351b76242a8c2b3c2310ab88b1600b1a8cbe121001a4cf0483d5 not found: ID does not exist"
Jul 24 13:37:52.122901 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.122876    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:52.127384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.127359    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca810d9a-675ccd59b5-hnvrc"]
Jul 24 13:37:52.128809 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.128790    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dvp9\" (UniqueName: \"kubernetes.io/projected/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-kube-api-access-4dvp9\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:52.128884 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.128814    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:52.128884 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:52.128829    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1fe53a1-4912-4f12-9a3a-2f53aba589e4-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:37:53.340572 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:53.340541    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" path="/var/lib/kubelet/pods/c1fe53a1-4912-4f12-9a3a-2f53aba589e4/volumes"
Jul 24 13:37:56.320512 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:56.320476    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"]
Jul 24 13:37:56.322843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:56.322817    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca810d9a-58nlg"]
Jul 24 13:37:57.341221 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:37:57.341178    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b0b353-7a85-4bdb-a259-f3214356dcfd" path="/var/lib/kubelet/pods/96b0b353-7a85-4bdb-a259-f3214356dcfd/volumes"
Jul 24 13:38:00.843475 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.843432    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:00.844203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844144    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" containerName="maas-api"
Jul 24 13:38:00.844203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844201    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" containerName="maas-api"
Jul 24 13:38:00.844362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844229    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b0b353-7a85-4bdb-a259-f3214356dcfd" containerName="revoke-keys"
Jul 24 13:38:00.844362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844238    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b0b353-7a85-4bdb-a259-f3214356dcfd" containerName="revoke-keys"
Jul 24 13:38:00.844362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844343    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1fe53a1-4912-4f12-9a3a-2f53aba589e4" containerName="maas-api"
Jul 24 13:38:00.844362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.844361    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b0b353-7a85-4bdb-a259-f3214356dcfd" containerName="revoke-keys"
Jul 24 13:38:00.849320 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.849298    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:00.852375 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.852219    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:38:00.852525 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.852454    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a35f265b-serving-cert\""
Jul 24 13:38:00.853800 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.853773    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:00.901804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.901774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:00.901804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.901809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:00.902038 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:00.901863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhwf\" (UniqueName: \"kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.002912 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.002877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhwf\" (UniqueName: \"kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.003120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.002984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.003120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.003015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.003242 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:38:01.003147    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-serving-cert: secret "maas-api-e2e-derive-a35f265b-serving-cert" not found
Jul 24 13:38:01.003242 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:38:01.003210    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls podName:17b1830f-b79e-4de0-b496-73d7e0c10f90 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:01.503188911 +0000 UTC m=+1700.727672221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls") pod "maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" (UID: "17b1830f-b79e-4de0-b496-73d7e0c10f90") : secret "maas-api-e2e-derive-a35f265b-serving-cert" not found
Jul 24 13:38:01.003648 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.003627    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.011388 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.011365    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhwf\" (UniqueName: \"kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.508344 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.508302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.510851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.510830    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") pod \"maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.560776 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.560745    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:01.563966 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.563925    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.566968 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.566925    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a35f265b-dockercfg-7w6jb\""
Jul 24 13:38:01.567089 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.567029    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a35f265b\""
Jul 24 13:38:01.577032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.577009    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:01.609471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.609438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.609694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.609532    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh84\" (UniqueName: \"kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.710436 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.710403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh84\" (UniqueName: \"kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.710602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.710502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.711278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.711261    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.718835 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.718804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh84\" (UniqueName: \"kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84\") pod \"payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.762760 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.762682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:01.874440 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.874407    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:01.897462 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:01.897437    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:01.899752 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:38:01.899730    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b1830f_b79e_4de0_b496_73d7e0c10f90.slice/crio-d2a25fed80fcbde051ffe0b83c6121dd3085ac797293e9b13bb30264e2e1d4b9 WatchSource:0}: Error finding container d2a25fed80fcbde051ffe0b83c6121dd3085ac797293e9b13bb30264e2e1d4b9: Status 404 returned error can't find the container with id d2a25fed80fcbde051ffe0b83c6121dd3085ac797293e9b13bb30264e2e1d4b9
Jul 24 13:38:02.008080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.008057    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:02.010805 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:38:02.010777    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c3a260_16e0_44df_8af5_3e80a0b298b8.slice/crio-af3823da443d9bf97b50a615fb7059b26ab6177d3787f7f31fe72a0d74b1350d WatchSource:0}: Error finding container af3823da443d9bf97b50a615fb7059b26ab6177d3787f7f31fe72a0d74b1350d: Status 404 returned error can't find the container with id af3823da443d9bf97b50a615fb7059b26ab6177d3787f7f31fe72a0d74b1350d
Jul 24 13:38:02.149099 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.149057    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" event={"ID":"36c3a260-16e0-44df-8af5-3e80a0b298b8","Type":"ContainerStarted","Data":"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"}
Jul 24 13:38:02.149270 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.149233    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:02.149359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.149290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" event={"ID":"36c3a260-16e0-44df-8af5-3e80a0b298b8","Type":"ContainerStarted","Data":"af3823da443d9bf97b50a615fb7059b26ab6177d3787f7f31fe72a0d74b1350d"}
Jul 24 13:38:02.150292 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.150270    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" event={"ID":"17b1830f-b79e-4de0-b496-73d7e0c10f90","Type":"ContainerStarted","Data":"d2a25fed80fcbde051ffe0b83c6121dd3085ac797293e9b13bb30264e2e1d4b9"}
Jul 24 13:38:02.164980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:02.164907    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" podStartSLOduration=1.1648944 podStartE2EDuration="1.1648944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:02.163926684 +0000 UTC m=+1701.388410023" watchObservedRunningTime="2026-07-24 13:38:02.1648944 +0000 UTC m=+1701.389377727"
Jul 24 13:38:03.156915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:03.156873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" event={"ID":"17b1830f-b79e-4de0-b496-73d7e0c10f90","Type":"ContainerStarted","Data":"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"}
Jul 24 13:38:03.157399 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:03.157118    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:03.172675 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:03.172622    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" podStartSLOduration=2.772301343 podStartE2EDuration="3.172602594s" totalImagesPullingTime="400.301251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:03.172090621 +0000 UTC m=+1702.396573950" watchObservedRunningTime="2026-07-24 13:38:03.172602594 +0000 UTC m=+1702.397085923"
Jul 24 13:38:09.167450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:09.167420    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:13.158797 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:13.158767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:25.512080 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.512046    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"]
Jul 24 13:38:25.515481 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.515465    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:25.517640 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.517613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:38:25.521489 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.521470    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"]
Jul 24 13:38:25.529839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.529816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2\" (UID: \"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:25.630472 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.630435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2\" (UID: \"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:25.631077 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.631059    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2\" (UID: \"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:25.828686 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.828655    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:25.963364 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:25.963334    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"]
Jul 24 13:38:25.965854 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:38:25.965831    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88d1112_b8bb_4a52_8eff_c3f7d566d0c8.slice/crio-69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442 WatchSource:0}: Error finding container 69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442: Status 404 returned error can't find the container with id 69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442
Jul 24 13:38:26.279515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:26.279480    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2" event={"ID":"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8","Type":"ContainerStarted","Data":"9a4564c7fbaaf3ed5ee6f56b946d2f939ed481c8afa8b2a7018f933a5d8c5e0e"}
Jul 24 13:38:26.279664 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:26.279538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2" event={"ID":"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8","Type":"ContainerStarted","Data":"69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442"}
Jul 24 13:38:26.296015 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:26.295971    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2" podStartSLOduration=1.295956506 podStartE2EDuration="1.295956506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:26.293538226 +0000 UTC m=+1725.518021555" watchObservedRunningTime="2026-07-24 13:38:26.295956506 +0000 UTC m=+1725.520439833"
Jul 24 13:38:27.285238 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:27.285205    2556 generic.go:356] "Generic (PLEG): container finished" podID="c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" containerID="9a4564c7fbaaf3ed5ee6f56b946d2f939ed481c8afa8b2a7018f933a5d8c5e0e" exitCode=0
Jul 24 13:38:27.285616 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:27.285276    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2" event={"ID":"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8","Type":"ContainerDied","Data":"9a4564c7fbaaf3ed5ee6f56b946d2f939ed481c8afa8b2a7018f933a5d8c5e0e"}
Jul 24 13:38:28.416720 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:28.416696    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:28.457305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:28.457279    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca\") pod \"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8\" (UID: \"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8\") "
Jul 24 13:38:28.457647 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:28.457626    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" (UID: "c88d1112-b8bb-4a52-8eff-c3f7d566d0c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:28.558738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:28.558662    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:29.296763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:29.296735    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"
Jul 24 13:38:29.296975 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:29.296730    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2" event={"ID":"c88d1112-b8bb-4a52-8eff-c3f7d566d0c8","Type":"ContainerDied","Data":"69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442"}
Jul 24 13:38:29.296975 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:29.296853    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fb085813af3ceb7227099f78f75161361f482aa1d1c3fb680ce5f2cff80442"
Jul 24 13:38:35.954318 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:35.954281    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:35.954739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:35.954602    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" podUID="17b1830f-b79e-4de0-b496-73d7e0c10f90" containerName="maas-api" containerID="cri-o://da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7" gracePeriod=30
Jul 24 13:38:36.017367 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.017335    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:36.017595 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.017546    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" podUID="36c3a260-16e0-44df-8af5-3e80a0b298b8" containerName="payload-pre-processing" containerID="cri-o://168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176" gracePeriod=30
Jul 24 13:38:36.277063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.277039    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:36.280429 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.280410    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:36.329396 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329362    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume\") pod \"36c3a260-16e0-44df-8af5-3e80a0b298b8\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") "
Jul 24 13:38:36.329565 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329444    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phh84\" (UniqueName: \"kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84\") pod \"36c3a260-16e0-44df-8af5-3e80a0b298b8\" (UID: \"36c3a260-16e0-44df-8af5-3e80a0b298b8\") "
Jul 24 13:38:36.329565 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329478    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") pod \"17b1830f-b79e-4de0-b496-73d7e0c10f90\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") "
Jul 24 13:38:36.329565 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329496    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca\") pod \"17b1830f-b79e-4de0-b496-73d7e0c10f90\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") "
Jul 24 13:38:36.329565 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329542    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhwf\" (UniqueName: \"kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf\") pod \"17b1830f-b79e-4de0-b496-73d7e0c10f90\" (UID: \"17b1830f-b79e-4de0-b496-73d7e0c10f90\") "
Jul 24 13:38:36.329886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329857    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36c3a260-16e0-44df-8af5-3e80a0b298b8" (UID: "36c3a260-16e0-44df-8af5-3e80a0b298b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:36.329988 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.329923    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17b1830f-b79e-4de0-b496-73d7e0c10f90" (UID: "17b1830f-b79e-4de0-b496-73d7e0c10f90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:36.332023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.331989    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf" (OuterVolumeSpecName: "kube-api-access-vhhwf") pod "17b1830f-b79e-4de0-b496-73d7e0c10f90" (UID: "17b1830f-b79e-4de0-b496-73d7e0c10f90"). InnerVolumeSpecName "kube-api-access-vhhwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:38:36.332139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.332072    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17b1830f-b79e-4de0-b496-73d7e0c10f90" (UID: "17b1830f-b79e-4de0-b496-73d7e0c10f90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:38:36.332139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.332077    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84" (OuterVolumeSpecName: "kube-api-access-phh84") pod "36c3a260-16e0-44df-8af5-3e80a0b298b8" (UID: "36c3a260-16e0-44df-8af5-3e80a0b298b8"). InnerVolumeSpecName "kube-api-access-phh84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:38:36.333629 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.333607    2556 generic.go:356] "Generic (PLEG): container finished" podID="17b1830f-b79e-4de0-b496-73d7e0c10f90" containerID="da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7" exitCode=0
Jul 24 13:38:36.333740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.333647    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"
Jul 24 13:38:36.333740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.333686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" event={"ID":"17b1830f-b79e-4de0-b496-73d7e0c10f90","Type":"ContainerDied","Data":"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"}
Jul 24 13:38:36.333740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.333740    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x" event={"ID":"17b1830f-b79e-4de0-b496-73d7e0c10f90","Type":"ContainerDied","Data":"d2a25fed80fcbde051ffe0b83c6121dd3085ac797293e9b13bb30264e2e1d4b9"}
Jul 24 13:38:36.333872 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.333767    2556 scope.go:117] "RemoveContainer" containerID="da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"
Jul 24 13:38:36.335101 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.335081    2556 generic.go:356] "Generic (PLEG): container finished" podID="36c3a260-16e0-44df-8af5-3e80a0b298b8" containerID="168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176" exitCode=0
Jul 24 13:38:36.335203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.335143    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"
Jul 24 13:38:36.335203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.335150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" event={"ID":"36c3a260-16e0-44df-8af5-3e80a0b298b8","Type":"ContainerDied","Data":"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"}
Jul 24 13:38:36.335203 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.335184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l" event={"ID":"36c3a260-16e0-44df-8af5-3e80a0b298b8","Type":"ContainerDied","Data":"af3823da443d9bf97b50a615fb7059b26ab6177d3787f7f31fe72a0d74b1350d"}
Jul 24 13:38:36.343215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.343192    2556 scope.go:117] "RemoveContainer" containerID="da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"
Jul 24 13:38:36.343484 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:38:36.343464    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7\": container with ID starting with da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7 not found: ID does not exist" containerID="da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"
Jul 24 13:38:36.343564 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.343489    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7"} err="failed to get container status \"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7\": rpc error: code = NotFound desc = could not find container \"da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7\": container with ID starting with da468250863ecca68e0d4577e44dc275fc1e485ee0d0a277de91ce3e97bf1eb7 not found: ID does not exist"
Jul 24 13:38:36.343564 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.343505    2556 scope.go:117] "RemoveContainer" containerID="168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"
Jul 24 13:38:36.351449 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.351430    2556 scope.go:117] "RemoveContainer" containerID="168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"
Jul 24 13:38:36.351690 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:38:36.351671    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176\": container with ID starting with 168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176 not found: ID does not exist" containerID="168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"
Jul 24 13:38:36.351737 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.351696    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176"} err="failed to get container status \"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176\": rpc error: code = NotFound desc = could not find container \"168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176\": container with ID starting with 168c7750dcb6f58614a500948196eb48c7f52f23a26a8634187eb966cfa3d176 not found: ID does not exist"
Jul 24 13:38:36.357677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.357658    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:36.363394 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.363375    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a35f265b-6c4d474f97-dcb8x"]
Jul 24 13:38:36.374752 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.374727    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:36.380315 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.380296    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a35f265b-56558d849b-gs98l"]
Jul 24 13:38:36.431073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.431048    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh84\" (UniqueName: \"kubernetes.io/projected/36c3a260-16e0-44df-8af5-3e80a0b298b8-kube-api-access-phh84\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:36.431073 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.431070    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17b1830f-b79e-4de0-b496-73d7e0c10f90-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:36.431243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.431147    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17b1830f-b79e-4de0-b496-73d7e0c10f90-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:36.431301 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.431247    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhwf\" (UniqueName: \"kubernetes.io/projected/17b1830f-b79e-4de0-b496-73d7e0c10f90-kube-api-access-vhhwf\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:36.431301 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:36.431277    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3a260-16e0-44df-8af5-3e80a0b298b8-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:38:37.340794 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:37.340759    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b1830f-b79e-4de0-b496-73d7e0c10f90" path="/var/lib/kubelet/pods/17b1830f-b79e-4de0-b496-73d7e0c10f90/volumes"
Jul 24 13:38:37.341274 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:37.341250    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c3a260-16e0-44df-8af5-3e80a0b298b8" path="/var/lib/kubelet/pods/36c3a260-16e0-44df-8af5-3e80a0b298b8/volumes"
Jul 24 13:38:40.591485 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:40.591451    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"]
Jul 24 13:38:40.593878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:40.593852    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a35f265b-w4wd2"]
Jul 24 13:38:41.340422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:38:41.340389    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" path="/var/lib/kubelet/pods/c88d1112-b8bb-4a52-8eff-c3f7d566d0c8/volumes"
Jul 24 13:39:41.283857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:39:41.283827    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:39:41.286353 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:39:41.286332    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:40:22.267629 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.267593    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.267995    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" containerName="revoke-keys"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268008    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" containerName="revoke-keys"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268022    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b1830f-b79e-4de0-b496-73d7e0c10f90" containerName="maas-api"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268027    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b1830f-b79e-4de0-b496-73d7e0c10f90" containerName="maas-api"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268036    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c3a260-16e0-44df-8af5-3e80a0b298b8" containerName="payload-pre-processing"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268043    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c3a260-16e0-44df-8af5-3e80a0b298b8" containerName="payload-pre-processing"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268106    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c3a260-16e0-44df-8af5-3e80a0b298b8" containerName="payload-pre-processing"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268112    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b1830f-b79e-4de0-b496-73d7e0c10f90" containerName="maas-api"
Jul 24 13:40:22.268144 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.268121    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c88d1112-b8bb-4a52-8eff-c3f7d566d0c8" containerName="revoke-keys"
Jul 24 13:40:22.271099 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.271084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.273461 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.273440    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bcb704f0-serving-cert\""
Jul 24 13:40:22.273589 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.273467    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:40:22.282967 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.282924    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:22.407415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.407377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfxt\" (UniqueName: \"kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.407589 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.407440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.407589 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.407537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.508624 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.508591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.508793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.508664    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.508793 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:40:22.508740    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-serving-cert: secret "maas-api-e2e-mt-bcb704f0-serving-cert" not found
Jul 24 13:40:22.508890 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:40:22.508823    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls podName:e8eef8c4-ec2e-45f4-b0ff-4196152efbaa nodeName:}" failed. No retries permitted until 2026-07-24 13:40:23.008799207 +0000 UTC m=+1842.233282532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls") pod "maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" (UID: "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa") : secret "maas-api-e2e-mt-bcb704f0-serving-cert" not found
Jul 24 13:40:22.508982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.508888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfxt\" (UniqueName: \"kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.509368 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.509349    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:22.520656 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:22.520602    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfxt\" (UniqueName: \"kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:23.013581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.013545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:23.016109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.016088    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") pod \"maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:23.079982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.079948    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:23.083447 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.083431    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.085959 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.085911    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bcb704f0-dockercfg-7wswd\""
Jul 24 13:40:23.086074 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.086033    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bcb704f0\""
Jul 24 13:40:23.099857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.099832    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:23.114748 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.114725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.114853 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.114778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pd56\" (UniqueName: \"kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.184158 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.184132    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:23.215455 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.215419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.215588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.215509    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pd56\" (UniqueName: \"kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.216300 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.216283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.224973 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.224925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pd56\" (UniqueName: \"kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56\") pod \"payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.315595 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.315571    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:23.318129 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:40:23.318101    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8eef8c4_ec2e_45f4_b0ff_4196152efbaa.slice/crio-ae08948c138ede62395f18d36b584b6d2f87984baea1725531465ffd3aee83c9 WatchSource:0}: Error finding container ae08948c138ede62395f18d36b584b6d2f87984baea1725531465ffd3aee83c9: Status 404 returned error can't find the container with id ae08948c138ede62395f18d36b584b6d2f87984baea1725531465ffd3aee83c9
Jul 24 13:40:23.393510 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.393472    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.525571 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.525534    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:23.528130 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:40:23.528103    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0826b6ca_59a3_4220_838b_eee684cf0adf.slice/crio-f52dbbc30ffe731b06b458ba664486aa6a917dafb44bdeccbb35600ce9d8548c WatchSource:0}: Error finding container f52dbbc30ffe731b06b458ba664486aa6a917dafb44bdeccbb35600ce9d8548c: Status 404 returned error can't find the container with id f52dbbc30ffe731b06b458ba664486aa6a917dafb44bdeccbb35600ce9d8548c
Jul 24 13:40:23.861181 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.861144    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" event={"ID":"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa","Type":"ContainerStarted","Data":"ae08948c138ede62395f18d36b584b6d2f87984baea1725531465ffd3aee83c9"}
Jul 24 13:40:23.862692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.862666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" event={"ID":"0826b6ca-59a3-4220-838b-eee684cf0adf","Type":"ContainerStarted","Data":"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"}
Jul 24 13:40:23.862806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.862716    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" event={"ID":"0826b6ca-59a3-4220-838b-eee684cf0adf","Type":"ContainerStarted","Data":"f52dbbc30ffe731b06b458ba664486aa6a917dafb44bdeccbb35600ce9d8548c"}
Jul 24 13:40:23.862806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.862756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:23.881710 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:23.881643    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" podStartSLOduration=0.881629017 podStartE2EDuration="881.629017ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:23.880599723 +0000 UTC m=+1843.105083050" watchObservedRunningTime="2026-07-24 13:40:23.881629017 +0000 UTC m=+1843.106112344"
Jul 24 13:40:24.870092 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:24.870049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" event={"ID":"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa","Type":"ContainerStarted","Data":"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"}
Jul 24 13:40:24.870563 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:24.870200    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:24.892741 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:24.892692    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" podStartSLOduration=2.390292674 podStartE2EDuration="2.892672192s" totalImagesPullingTime="502.379518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:24.890389496 +0000 UTC m=+1844.114872826" watchObservedRunningTime="2026-07-24 13:40:24.892672192 +0000 UTC m=+1844.117155519"
Jul 24 13:40:30.880418 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:30.880387    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:34.141553 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.141513    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"]
Jul 24 13:40:34.145300 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.145278    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:34.147546 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.147525    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:40:34.151552 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.151529    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"]
Jul 24 13:40:34.216583 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.216539    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc\" (UID: \"1ce0e819-ecb2-4987-9f14-762e9635fe55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:34.317067 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.317031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc\" (UID: \"1ce0e819-ecb2-4987-9f14-762e9635fe55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:34.317660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.317641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc\" (UID: \"1ce0e819-ecb2-4987-9f14-762e9635fe55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:34.456860 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.456789    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:34.599540 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.599508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"]
Jul 24 13:40:34.602801 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:40:34.602775    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce0e819_ecb2_4987_9f14_762e9635fe55.slice/crio-7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6 WatchSource:0}: Error finding container 7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6: Status 404 returned error can't find the container with id 7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6
Jul 24 13:40:34.872502 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.872475    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:34.919402 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.919368    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc" event={"ID":"1ce0e819-ecb2-4987-9f14-762e9635fe55","Type":"ContainerStarted","Data":"53461f874c99056d3e34652ef1af6c4e6dff26428c662544c23630ba2024a25f"}
Jul 24 13:40:34.919551 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.919432    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc" event={"ID":"1ce0e819-ecb2-4987-9f14-762e9635fe55","Type":"ContainerStarted","Data":"7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6"}
Jul 24 13:40:34.934422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:34.934377    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc" podStartSLOduration=0.934361809 podStartE2EDuration="934.361809ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:34.932895506 +0000 UTC m=+1854.157378835" watchObservedRunningTime="2026-07-24 13:40:34.934361809 +0000 UTC m=+1854.158845135"
Jul 24 13:40:35.927021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:35.926985    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ce0e819-ecb2-4987-9f14-762e9635fe55" containerID="53461f874c99056d3e34652ef1af6c4e6dff26428c662544c23630ba2024a25f" exitCode=0
Jul 24 13:40:35.927396 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:35.927077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc" event={"ID":"1ce0e819-ecb2-4987-9f14-762e9635fe55","Type":"ContainerDied","Data":"53461f874c99056d3e34652ef1af6c4e6dff26428c662544c23630ba2024a25f"}
Jul 24 13:40:37.068778 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.068754    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:37.141741 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.141705    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca\") pod \"1ce0e819-ecb2-4987-9f14-762e9635fe55\" (UID: \"1ce0e819-ecb2-4987-9f14-762e9635fe55\") "
Jul 24 13:40:37.142142 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.142120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ce0e819-ecb2-4987-9f14-762e9635fe55" (UID: "1ce0e819-ecb2-4987-9f14-762e9635fe55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:40:37.243369 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.243283    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce0e819-ecb2-4987-9f14-762e9635fe55-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:37.944251 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.944217    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"
Jul 24 13:40:37.944427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.944217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc" event={"ID":"1ce0e819-ecb2-4987-9f14-762e9635fe55","Type":"ContainerDied","Data":"7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6"}
Jul 24 13:40:37.944427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:37.944366    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5a15ac12cce2f5cb0fef53da8d47bd96d6182c8b148da569ecbb707cbef0d6"
Jul 24 13:40:44.649171 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.649139    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:44.649559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.649367    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" podUID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" containerName="maas-api" containerID="cri-o://24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7" gracePeriod=30
Jul 24 13:40:44.706218 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.706187    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:44.706448 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.706417    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerName="payload-pre-processing" containerID="cri-o://dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53" gracePeriod=30
Jul 24 13:40:44.870520 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.870475    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.45:9004: connect: connection refused"
Jul 24 13:40:44.969517 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.969489    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:44.972743 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.972722    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:44.979886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.979862    2556 generic.go:356] "Generic (PLEG): container finished" podID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerID="dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53" exitCode=0
Jul 24 13:40:44.980008 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.979896    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"
Jul 24 13:40:44.980008 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.979960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" event={"ID":"0826b6ca-59a3-4220-838b-eee684cf0adf","Type":"ContainerDied","Data":"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"}
Jul 24 13:40:44.980126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.980021    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2" event={"ID":"0826b6ca-59a3-4220-838b-eee684cf0adf","Type":"ContainerDied","Data":"f52dbbc30ffe731b06b458ba664486aa6a917dafb44bdeccbb35600ce9d8548c"}
Jul 24 13:40:44.980126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.980045    2556 scope.go:117] "RemoveContainer" containerID="dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"
Jul 24 13:40:44.981269 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.981249    2556 generic.go:356] "Generic (PLEG): container finished" podID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" containerID="24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7" exitCode=0
Jul 24 13:40:44.981366 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.981288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" event={"ID":"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa","Type":"ContainerDied","Data":"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"}
Jul 24 13:40:44.981366 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.981323    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97" event={"ID":"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa","Type":"ContainerDied","Data":"ae08948c138ede62395f18d36b584b6d2f87984baea1725531465ffd3aee83c9"}
Jul 24 13:40:44.981366 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.981290    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"
Jul 24 13:40:44.988860 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.988812    2556 scope.go:117] "RemoveContainer" containerID="dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"
Jul 24 13:40:44.989176 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:40:44.989134    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53\": container with ID starting with dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53 not found: ID does not exist" containerID="dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"
Jul 24 13:40:44.989284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.989180    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53"} err="failed to get container status \"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53\": rpc error: code = NotFound desc = could not find container \"dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53\": container with ID starting with dcfc8a4c77c4eeaa8e866a8e05f206deda1a58cd82075bcbc00790b7219e4f53 not found: ID does not exist"
Jul 24 13:40:44.989284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.989202    2556 scope.go:117] "RemoveContainer" containerID="24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"
Jul 24 13:40:44.998841 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.998812    2556 scope.go:117] "RemoveContainer" containerID="24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"
Jul 24 13:40:44.999116 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:40:44.999097    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7\": container with ID starting with 24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7 not found: ID does not exist" containerID="24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"
Jul 24 13:40:44.999183 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:44.999128    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7"} err="failed to get container status \"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7\": rpc error: code = NotFound desc = could not find container \"24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7\": container with ID starting with 24b01d94a71faba62cb86180ffee6ae4c954a3cac9b7b31629f5b2276eec15b7 not found: ID does not exist"
Jul 24 13:40:45.117448 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") pod \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") "
Jul 24 13:40:45.117630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117466    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pd56\" (UniqueName: \"kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56\") pod \"0826b6ca-59a3-4220-838b-eee684cf0adf\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") "
Jul 24 13:40:45.117630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117502    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfxt\" (UniqueName: \"kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt\") pod \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") "
Jul 24 13:40:45.117630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117531    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca\") pod \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\" (UID: \"e8eef8c4-ec2e-45f4-b0ff-4196152efbaa\") "
Jul 24 13:40:45.117630 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume\") pod \"0826b6ca-59a3-4220-838b-eee684cf0adf\" (UID: \"0826b6ca-59a3-4220-838b-eee684cf0adf\") "
Jul 24 13:40:45.117902 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117881    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" (UID: "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:40:45.118010 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.117984    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0826b6ca-59a3-4220-838b-eee684cf0adf" (UID: "0826b6ca-59a3-4220-838b-eee684cf0adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:40:45.119690 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.119670    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" (UID: "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:40:45.119742 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.119671    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt" (OuterVolumeSpecName: "kube-api-access-nvfxt") pod "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" (UID: "e8eef8c4-ec2e-45f4-b0ff-4196152efbaa"). InnerVolumeSpecName "kube-api-access-nvfxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:45.119742 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.119716    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56" (OuterVolumeSpecName: "kube-api-access-8pd56") pod "0826b6ca-59a3-4220-838b-eee684cf0adf" (UID: "0826b6ca-59a3-4220-838b-eee684cf0adf"). InnerVolumeSpecName "kube-api-access-8pd56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:40:45.218548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.218464    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:45.218548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.218492    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pd56\" (UniqueName: \"kubernetes.io/projected/0826b6ca-59a3-4220-838b-eee684cf0adf-kube-api-access-8pd56\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:45.218548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.218501    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfxt\" (UniqueName: \"kubernetes.io/projected/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-kube-api-access-nvfxt\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:45.218548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.218511    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:45.218548 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.218520    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0826b6ca-59a3-4220-838b-eee684cf0adf-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:40:45.302922 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.302887    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:45.307035 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.307009    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bcb704f0-547c86bcf6-cr7z2"]
Jul 24 13:40:45.316694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.316671    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:45.320532 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.320511    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bcb704f0-65784dfbc7-p5t97"]
Jul 24 13:40:45.342012 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.341988    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" path="/var/lib/kubelet/pods/0826b6ca-59a3-4220-838b-eee684cf0adf/volumes"
Jul 24 13:40:45.342373 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:45.342360    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" path="/var/lib/kubelet/pods/e8eef8c4-ec2e-45f4-b0ff-4196152efbaa/volumes"
Jul 24 13:40:49.224819 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:49.224780    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"]
Jul 24 13:40:49.228118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:49.228095    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bcb704f0-q75wc"]
Jul 24 13:40:49.340602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:40:49.340570    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce0e819-ecb2-4987-9f14-762e9635fe55" path="/var/lib/kubelet/pods/1ce0e819-ecb2-4987-9f14-762e9635fe55/volumes"
Jul 24 13:41:19.289225 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289175    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289684    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" containerName="maas-api"
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289700    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" containerName="maas-api"
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289728    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce0e819-ecb2-4987-9f14-762e9635fe55" containerName="revoke-keys"
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289737    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce0e819-ecb2-4987-9f14-762e9635fe55" containerName="revoke-keys"
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289755    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerName="payload-pre-processing"
Jul 24 13:41:19.289834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289764    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerName="payload-pre-processing"
Jul 24 13:41:19.290200 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289850    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0826b6ca-59a3-4220-838b-eee684cf0adf" containerName="payload-pre-processing"
Jul 24 13:41:19.290200 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289862    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce0e819-ecb2-4987-9f14-762e9635fe55" containerName="revoke-keys"
Jul 24 13:41:19.290200 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.289871    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8eef8c4-ec2e-45f4-b0ff-4196152efbaa" containerName="maas-api"
Jul 24 13:41:19.293017 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.292994    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.296544 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.296512    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2cdea471-serving-cert\""
Jul 24 13:41:19.296660 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.296574    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:41:19.302370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.302343    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:41:19.323336 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.323305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.323492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.323364    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsmx\" (UniqueName: \"kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.323492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.323401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.424629 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.424592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.424838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.424650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsmx\" (UniqueName: \"kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.424838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.424673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.424838 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:41:19.424754    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-serving-cert: secret "maas-api-e2e-mt-2cdea471-serving-cert" not found
Jul 24 13:41:19.424838 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:41:19.424835    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls podName:d9292c82-6b7a-45a9-90e1-a36980d47a42 nodeName:}" failed. No retries permitted until 2026-07-24 13:41:19.924818009 +0000 UTC m=+1899.149301314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls") pod "maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" (UID: "d9292c82-6b7a-45a9-90e1-a36980d47a42") : secret "maas-api-e2e-mt-2cdea471-serving-cert" not found
Jul 24 13:41:19.425353 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.425335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.435714 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.435694    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsmx\" (UniqueName: \"kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.929966 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.929918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.932513 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.932496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") pod \"maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:19.996556 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:19.996518    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:41:20.000038 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.000021    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.009998 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.009977    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2cdea471\""
Jul 24 13:41:20.010121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.010067    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2cdea471-dockercfg-9z8zr\""
Jul 24 13:41:20.019557 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.019526    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:41:20.131175 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.131144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgztk\" (UniqueName: \"kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.131338 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.131239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.205155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.205068    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:20.231993 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.231965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.232156 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.232049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgztk\" (UniqueName: \"kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.232696 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.232671    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.240052 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.240027    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgztk\" (UniqueName: \"kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk\") pod \"payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.310484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.310449    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:20.344234 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.344207    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:41:20.346824 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:41:20.346790    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9292c82_6b7a_45a9_90e1_a36980d47a42.slice/crio-6dac872f016141458e21638257222fc4ab93c02c9f3bfb23a870db42536d0d67 WatchSource:0}: Error finding container 6dac872f016141458e21638257222fc4ab93c02c9f3bfb23a870db42536d0d67: Status 404 returned error can't find the container with id 6dac872f016141458e21638257222fc4ab93c02c9f3bfb23a870db42536d0d67
Jul 24 13:41:20.348110 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.348093    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:41:20.452588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:20.452562    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:41:20.454988 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:41:20.454957    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc916e23d_5c1e_4761_86e0_94ee788e7f1a.slice/crio-5ad70fdd6c68a40575eef40d14ec3e5b20c6d598154a9f662d09259663cb392d WatchSource:0}: Error finding container 5ad70fdd6c68a40575eef40d14ec3e5b20c6d598154a9f662d09259663cb392d: Status 404 returned error can't find the container with id 5ad70fdd6c68a40575eef40d14ec3e5b20c6d598154a9f662d09259663cb392d
Jul 24 13:41:21.157724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.157688    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" event={"ID":"c916e23d-5c1e-4761-86e0-94ee788e7f1a","Type":"ContainerStarted","Data":"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"}
Jul 24 13:41:21.157915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.157770    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:21.157915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.157797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" event={"ID":"c916e23d-5c1e-4761-86e0-94ee788e7f1a","Type":"ContainerStarted","Data":"5ad70fdd6c68a40575eef40d14ec3e5b20c6d598154a9f662d09259663cb392d"}
Jul 24 13:41:21.159224 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.159201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" event={"ID":"d9292c82-6b7a-45a9-90e1-a36980d47a42","Type":"ContainerStarted","Data":"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"}
Jul 24 13:41:21.159303 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.159240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" event={"ID":"d9292c82-6b7a-45a9-90e1-a36980d47a42","Type":"ContainerStarted","Data":"6dac872f016141458e21638257222fc4ab93c02c9f3bfb23a870db42536d0d67"}
Jul 24 13:41:21.159303 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.159258    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:21.178596 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.178543    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" podStartSLOduration=2.178525635 podStartE2EDuration="2.178525635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:21.178512713 +0000 UTC m=+1900.402996040" watchObservedRunningTime="2026-07-24 13:41:21.178525635 +0000 UTC m=+1900.403008964"
Jul 24 13:41:21.199228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:21.199166    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" podStartSLOduration=1.859192875 podStartE2EDuration="2.199140516s" totalImagesPullingTime="339.947641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:21.195624834 +0000 UTC m=+1900.420108160" watchObservedRunningTime="2026-07-24 13:41:21.199140516 +0000 UTC m=+1900.423623844"
Jul 24 13:41:27.170642 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:27.170610    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:41:32.167458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:32.167424    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:41:58.484062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.484029    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"]
Jul 24 13:41:58.489003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.488977    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:41:58.491555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.491532    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:41:58.494281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.494254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"]
Jul 24 13:41:58.579238 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.579196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc\" (UID: \"3dfac727-5657-4e03-b8df-5f7c69e4956d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:41:58.680749 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.680709    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc\" (UID: \"3dfac727-5657-4e03-b8df-5f7c69e4956d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:41:58.681403 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.681378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc\" (UID: \"3dfac727-5657-4e03-b8df-5f7c69e4956d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:41:58.801498 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.801462    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:41:58.931159 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:58.931129    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"]
Jul 24 13:41:58.933676 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:41:58.933650    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfac727_5657_4e03_b8df_5f7c69e4956d.slice/crio-71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46 WatchSource:0}: Error finding container 71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46: Status 404 returned error can't find the container with id 71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46
Jul 24 13:41:59.358446 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:59.358409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc" event={"ID":"3dfac727-5657-4e03-b8df-5f7c69e4956d","Type":"ContainerStarted","Data":"905ec63cab4c6523e7a2c5a8adf644ca00f0d3ae3647a60bf5e1b761095e0c38"}
Jul 24 13:41:59.358614 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:59.358468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc" event={"ID":"3dfac727-5657-4e03-b8df-5f7c69e4956d","Type":"ContainerStarted","Data":"71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46"}
Jul 24 13:41:59.374350 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:41:59.374303    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc" podStartSLOduration=1.374288735 podStartE2EDuration="1.374288735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:59.372503776 +0000 UTC m=+1938.596987103" watchObservedRunningTime="2026-07-24 13:41:59.374288735 +0000 UTC m=+1938.598772062"
Jul 24 13:42:00.364794 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:00.364754    2556 generic.go:356] "Generic (PLEG): container finished" podID="3dfac727-5657-4e03-b8df-5f7c69e4956d" containerID="905ec63cab4c6523e7a2c5a8adf644ca00f0d3ae3647a60bf5e1b761095e0c38" exitCode=0
Jul 24 13:42:00.365312 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:00.364842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc" event={"ID":"3dfac727-5657-4e03-b8df-5f7c69e4956d","Type":"ContainerDied","Data":"905ec63cab4c6523e7a2c5a8adf644ca00f0d3ae3647a60bf5e1b761095e0c38"}
Jul 24 13:42:01.504015 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:01.503984    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:42:01.601135 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:01.601096    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca\") pod \"3dfac727-5657-4e03-b8df-5f7c69e4956d\" (UID: \"3dfac727-5657-4e03-b8df-5f7c69e4956d\") "
Jul 24 13:42:01.601426 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:01.601401    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3dfac727-5657-4e03-b8df-5f7c69e4956d" (UID: "3dfac727-5657-4e03-b8df-5f7c69e4956d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:01.702052 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:01.701926    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfac727-5657-4e03-b8df-5f7c69e4956d-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:02.376063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:02.376035    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"
Jul 24 13:42:02.376243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:02.376035    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc" event={"ID":"3dfac727-5657-4e03-b8df-5f7c69e4956d","Type":"ContainerDied","Data":"71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46"}
Jul 24 13:42:02.376243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:02.376164    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a11499328848b232bb078ea6855f7720e6c64d77214e06e68e1e1a12af3d46"
Jul 24 13:42:13.999243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:13.999198    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:42:13.999669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:13.999453    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" podUID="d9292c82-6b7a-45a9-90e1-a36980d47a42" containerName="maas-api" containerID="cri-o://76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e" gracePeriod=30
Jul 24 13:42:14.043121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.043087    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:42:14.043370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.043326    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" podUID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" containerName="payload-pre-processing" containerID="cri-o://50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e" gracePeriod=30
Jul 24 13:42:14.287342 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.287317    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:42:14.316699 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.316671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:42:14.419555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419518    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgztk\" (UniqueName: \"kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk\") pod \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") "
Jul 24 13:42:14.419555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419562    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") pod \"d9292c82-6b7a-45a9-90e1-a36980d47a42\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") "
Jul 24 13:42:14.419812 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzsmx\" (UniqueName: \"kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx\") pod \"d9292c82-6b7a-45a9-90e1-a36980d47a42\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") "
Jul 24 13:42:14.419812 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419614    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca\") pod \"d9292c82-6b7a-45a9-90e1-a36980d47a42\" (UID: \"d9292c82-6b7a-45a9-90e1-a36980d47a42\") "
Jul 24 13:42:14.419812 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419635    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume\") pod \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\" (UID: \"c916e23d-5c1e-4761-86e0-94ee788e7f1a\") "
Jul 24 13:42:14.419984 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.419963    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9292c82-6b7a-45a9-90e1-a36980d47a42" (UID: "d9292c82-6b7a-45a9-90e1-a36980d47a42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:14.420076 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.420047    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c916e23d-5c1e-4761-86e0-94ee788e7f1a" (UID: "c916e23d-5c1e-4761-86e0-94ee788e7f1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:14.421900 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.421874    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx" (OuterVolumeSpecName: "kube-api-access-zzsmx") pod "d9292c82-6b7a-45a9-90e1-a36980d47a42" (UID: "d9292c82-6b7a-45a9-90e1-a36980d47a42"). InnerVolumeSpecName "kube-api-access-zzsmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:14.422053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.421902    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9292c82-6b7a-45a9-90e1-a36980d47a42" (UID: "d9292c82-6b7a-45a9-90e1-a36980d47a42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:42:14.422053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.421915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk" (OuterVolumeSpecName: "kube-api-access-lgztk") pod "c916e23d-5c1e-4761-86e0-94ee788e7f1a" (UID: "c916e23d-5c1e-4761-86e0-94ee788e7f1a"). InnerVolumeSpecName "kube-api-access-lgztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:14.434277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.434250    2556 generic.go:356] "Generic (PLEG): container finished" podID="d9292c82-6b7a-45a9-90e1-a36980d47a42" containerID="76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e" exitCode=0
Jul 24 13:42:14.434385 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.434297    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"
Jul 24 13:42:14.434385 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.434331    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" event={"ID":"d9292c82-6b7a-45a9-90e1-a36980d47a42","Type":"ContainerDied","Data":"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"}
Jul 24 13:42:14.434385 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.434380    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8" event={"ID":"d9292c82-6b7a-45a9-90e1-a36980d47a42","Type":"ContainerDied","Data":"6dac872f016141458e21638257222fc4ab93c02c9f3bfb23a870db42536d0d67"}
Jul 24 13:42:14.434492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.434403    2556 scope.go:117] "RemoveContainer" containerID="76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"
Jul 24 13:42:14.435775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.435747    2556 generic.go:356] "Generic (PLEG): container finished" podID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" containerID="50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e" exitCode=0
Jul 24 13:42:14.435880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.435787    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"
Jul 24 13:42:14.435880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.435796    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" event={"ID":"c916e23d-5c1e-4761-86e0-94ee788e7f1a","Type":"ContainerDied","Data":"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"}
Jul 24 13:42:14.435880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.435829    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj" event={"ID":"c916e23d-5c1e-4761-86e0-94ee788e7f1a","Type":"ContainerDied","Data":"5ad70fdd6c68a40575eef40d14ec3e5b20c6d598154a9f662d09259663cb392d"}
Jul 24 13:42:14.444341 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.444319    2556 scope.go:117] "RemoveContainer" containerID="76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"
Jul 24 13:42:14.444618 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:42:14.444590    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e\": container with ID starting with 76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e not found: ID does not exist" containerID="76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"
Jul 24 13:42:14.444693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.444631    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e"} err="failed to get container status \"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e\": rpc error: code = NotFound desc = could not find container \"76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e\": container with ID starting with 76c0ab537074bbdf412dcc05bd667adff0ff5830e95c9832d89dc6fed887c79e not found: ID does not exist"
Jul 24 13:42:14.444693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.444653    2556 scope.go:117] "RemoveContainer" containerID="50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"
Jul 24 13:42:14.452685 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.452663    2556 scope.go:117] "RemoveContainer" containerID="50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"
Jul 24 13:42:14.452951 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:42:14.452912    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e\": container with ID starting with 50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e not found: ID does not exist" containerID="50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"
Jul 24 13:42:14.453015 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.452952    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e"} err="failed to get container status \"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e\": rpc error: code = NotFound desc = could not find container \"50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e\": container with ID starting with 50fda24d3dc04f3265d6687a8bd0376cb30335b73b6c0680bd8e4a55b196196e not found: ID does not exist"
Jul 24 13:42:14.458899 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.458875    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:42:14.465509 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.465487    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2cdea471-dfd59f84d-gpfdj"]
Jul 24 13:42:14.474902 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.474880    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:42:14.478677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.478657    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2cdea471-6dccf8dd5d-t6xv8"]
Jul 24 13:42:14.521048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.520962    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgztk\" (UniqueName: \"kubernetes.io/projected/c916e23d-5c1e-4761-86e0-94ee788e7f1a-kube-api-access-lgztk\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:14.521048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.520986    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9292c82-6b7a-45a9-90e1-a36980d47a42-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:14.521048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.520996    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzsmx\" (UniqueName: \"kubernetes.io/projected/d9292c82-6b7a-45a9-90e1-a36980d47a42-kube-api-access-zzsmx\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:14.521048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.521004    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9292c82-6b7a-45a9-90e1-a36980d47a42-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:14.521048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:14.521013    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c916e23d-5c1e-4761-86e0-94ee788e7f1a-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:42:15.341196 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:15.341166    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" path="/var/lib/kubelet/pods/c916e23d-5c1e-4761-86e0-94ee788e7f1a/volumes"
Jul 24 13:42:15.341593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:15.341527    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9292c82-6b7a-45a9-90e1-a36980d47a42" path="/var/lib/kubelet/pods/d9292c82-6b7a-45a9-90e1-a36980d47a42/volumes"
Jul 24 13:42:18.576504 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:18.576467    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"]
Jul 24 13:42:18.579015 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:18.578986    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2cdea471-4kdvc"]
Jul 24 13:42:19.340646 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:19.340610    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dfac727-5657-4e03-b8df-5f7c69e4956d" path="/var/lib/kubelet/pods/3dfac727-5657-4e03-b8df-5f7c69e4956d/volumes"
Jul 24 13:42:41.876348 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:42:41.876274    2556 scope.go:117] "RemoveContainer" containerID="cdb7d32c1836717742ddc2a54f8b2ee267a990af7722abbf9b439709012b547b"
Jul 24 13:43:41.904762 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:43:41.904728    2556 scope.go:117] "RemoveContainer" containerID="980dacd6bf876a468bd7ab7828f35290e235f7f6d32e0aa25f34e9c036370df7"
Jul 24 13:43:41.914754 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:43:41.914728    2556 scope.go:117] "RemoveContainer" containerID="0c51346d69f05a50a5a1420ce2804db9d1bf70baa43fc5f7cbdfce4ad5837240"
Jul 24 13:44:34.447140 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447109    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447444    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dfac727-5657-4e03-b8df-5f7c69e4956d" containerName="revoke-keys"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447455    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dfac727-5657-4e03-b8df-5f7c69e4956d" containerName="revoke-keys"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447471    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" containerName="payload-pre-processing"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447476    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" containerName="payload-pre-processing"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447487    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9292c82-6b7a-45a9-90e1-a36980d47a42" containerName="maas-api"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447492    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9292c82-6b7a-45a9-90e1-a36980d47a42" containerName="maas-api"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447549    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c916e23d-5c1e-4761-86e0-94ee788e7f1a" containerName="payload-pre-processing"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447557    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dfac727-5657-4e03-b8df-5f7c69e4956d" containerName="revoke-keys"
Jul 24 13:44:34.447703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.447564    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9292c82-6b7a-45a9-90e1-a36980d47a42" containerName="maas-api"
Jul 24 13:44:34.450653 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.450635    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.452820 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.452792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50b298-serving-cert\""
Jul 24 13:44:34.452956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.452859    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:44:34.461265 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.461243    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:44:34.577388 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.577352    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjd7\" (UniqueName: \"kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.577622 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.577410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.577622 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.577467    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.678155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.678111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjd7\" (UniqueName: \"kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.678379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.678171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.678379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.678197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.678379 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:44:34.678284    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-serving-cert: secret "maas-api-e2e-inf-a-50b298-serving-cert" not found
Jul 24 13:44:34.678379 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:44:34.678358    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls podName:2d748d26-9024-4168-944d-a0d7db283141 nodeName:}" failed. No retries permitted until 2026-07-24 13:44:35.178334745 +0000 UTC m=+2094.402818051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls") pod "maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" (UID: "2d748d26-9024-4168-944d-a0d7db283141") : secret "maas-api-e2e-inf-a-50b298-serving-cert" not found
Jul 24 13:44:34.678831 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.678812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:34.688575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:34.688554    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjd7\" (UniqueName: \"kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:35.182238 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:35.182197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:35.184836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:35.184803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:35.361433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:35.361397    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:35.494791 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:35.494717    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:44:35.497083 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:44:35.497058    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d748d26_9024_4168_944d_a0d7db283141.slice/crio-d1c35f229202f367d17aa0bd4c116a23662dcc92bb1636b1e40d91ab16892e38 WatchSource:0}: Error finding container d1c35f229202f367d17aa0bd4c116a23662dcc92bb1636b1e40d91ab16892e38: Status 404 returned error can't find the container with id d1c35f229202f367d17aa0bd4c116a23662dcc92bb1636b1e40d91ab16892e38
Jul 24 13:44:36.116315 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:36.116280    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" event={"ID":"2d748d26-9024-4168-944d-a0d7db283141","Type":"ContainerStarted","Data":"9bebc18fc3847c1236ed3c75ac42e7ba1df83bccc1b3e4b2d4978c9b05731283"}
Jul 24 13:44:36.116520 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:36.116334    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" event={"ID":"2d748d26-9024-4168-944d-a0d7db283141","Type":"ContainerStarted","Data":"d1c35f229202f367d17aa0bd4c116a23662dcc92bb1636b1e40d91ab16892e38"}
Jul 24 13:44:36.116520 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:36.116369    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:36.148693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:36.148646    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" podStartSLOduration=1.698139415 podStartE2EDuration="2.148625385s" totalImagesPullingTime="450.48597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:36.148231787 +0000 UTC m=+2095.372715125" watchObservedRunningTime="2026-07-24 13:44:36.148625385 +0000 UTC m=+2095.373108712"
Jul 24 13:44:41.314730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:41.314685    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:44:41.318140 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:41.318121    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:44:41.936045 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:41.936015    2556 scope.go:117] "RemoveContainer" containerID="9a4564c7fbaaf3ed5ee6f56b946d2f939ed481c8afa8b2a7018f933a5d8c5e0e"
Jul 24 13:44:42.126800 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.126772    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:44:42.342991 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.342962    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:44:42.347728 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.347707    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.350176 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.350151    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7918ee-serving-cert\""
Jul 24 13:44:42.359487 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.359467    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:44:42.446958 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.446904    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.447145 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.446998    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.447145 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.447055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnc5n\" (UniqueName: \"kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.547860 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.547821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.548120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.547916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.548120 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:44:42.547995    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-serving-cert: secret "maas-api-e2e-inf-b-7918ee-serving-cert" not found
Jul 24 13:44:42.548120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.548006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnc5n\" (UniqueName: \"kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.548120 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:44:42.548079    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls podName:a7695ef7-dd58-4329-b6e1-712053fbc1f1 nodeName:}" failed. No retries permitted until 2026-07-24 13:44:43.048057585 +0000 UTC m=+2102.272540892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls") pod "maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" (UID: "a7695ef7-dd58-4329-b6e1-712053fbc1f1") : secret "maas-api-e2e-inf-b-7918ee-serving-cert" not found
Jul 24 13:44:42.548576 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.548559    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:42.557016 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:42.556993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnc5n\" (UniqueName: \"kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:43.053227 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.053197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:43.055863 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.055838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:43.139484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.139449    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:44:43.142737 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.142721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.145020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.145001    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7918ee\""
Jul 24 13:44:43.145081 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.145070    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7918ee-dockercfg-km497\""
Jul 24 13:44:43.153886 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.153860    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:44:43.255684 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.255649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.255861 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.255699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p2m\" (UniqueName: \"kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.258585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.258555    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:43.356321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.356290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p2m\" (UniqueName: \"kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.356786 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.356377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.357006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.356985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.365174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.365151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p2m\" (UniqueName: \"kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m\") pod \"payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.398598 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.398483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:44:43.400677 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:44:43.400655    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7695ef7_dd58_4329_b6e1_712053fbc1f1.slice/crio-f568f394d6d48c7d1108dcf64bc15a861f2bcf3f3116aad3e6a72192bf8d1ffe WatchSource:0}: Error finding container f568f394d6d48c7d1108dcf64bc15a861f2bcf3f3116aad3e6a72192bf8d1ffe: Status 404 returned error can't find the container with id f568f394d6d48c7d1108dcf64bc15a861f2bcf3f3116aad3e6a72192bf8d1ffe
Jul 24 13:44:43.454712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.454673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:43.590402 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:43.590362    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:44:43.592869 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:44:43.592845    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31838071_57f0_4185_9241_1e84152ae76c.slice/crio-7929fcd83cb70c2406e02b51898cf3d6329eb9a75284986b72a9df52f5ab7c02 WatchSource:0}: Error finding container 7929fcd83cb70c2406e02b51898cf3d6329eb9a75284986b72a9df52f5ab7c02: Status 404 returned error can't find the container with id 7929fcd83cb70c2406e02b51898cf3d6329eb9a75284986b72a9df52f5ab7c02
Jul 24 13:44:44.157943 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.157893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" event={"ID":"31838071-57f0-4185-9241-1e84152ae76c","Type":"ContainerStarted","Data":"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"}
Jul 24 13:44:44.158143 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.157988    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" event={"ID":"31838071-57f0-4185-9241-1e84152ae76c","Type":"ContainerStarted","Data":"7929fcd83cb70c2406e02b51898cf3d6329eb9a75284986b72a9df52f5ab7c02"}
Jul 24 13:44:44.158143 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.158011    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:44:44.159492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.159466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" event={"ID":"a7695ef7-dd58-4329-b6e1-712053fbc1f1","Type":"ContainerStarted","Data":"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"}
Jul 24 13:44:44.159608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.159509    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:44.159608 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.159525    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" event={"ID":"a7695ef7-dd58-4329-b6e1-712053fbc1f1","Type":"ContainerStarted","Data":"f568f394d6d48c7d1108dcf64bc15a861f2bcf3f3116aad3e6a72192bf8d1ffe"}
Jul 24 13:44:44.180285 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.180237    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" podStartSLOduration=1.180224348 podStartE2EDuration="1.180224348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:44.177844899 +0000 UTC m=+2103.402328226" watchObservedRunningTime="2026-07-24 13:44:44.180224348 +0000 UTC m=+2103.404707674"
Jul 24 13:44:44.197650 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:44.197605    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" podStartSLOduration=1.8653376879999999 podStartE2EDuration="2.197592874s" totalImagesPullingTime="332.255186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:44.196769053 +0000 UTC m=+2103.421252379" watchObservedRunningTime="2026-07-24 13:44:44.197592874 +0000 UTC m=+2103.422076201"
Jul 24 13:44:50.170339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:50.170308    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:44:55.166733 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:44:55.166703    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:45:00.133823 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.133780    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"]
Jul 24 13:45:00.137457 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.137437    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"]
Jul 24 13:45:00.137587 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.137571    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"
Jul 24 13:45:00.139897 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.139874    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:45:00.140567 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.140552    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"]
Jul 24 13:45:00.140673 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.140659    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"
Jul 24 13:45:00.143730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.143705    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"
Jul 24 13:45:00.146153 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.146121    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"]
Jul 24 13:45:00.148333 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.148302    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"]
Jul 24 13:45:00.150190 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.150161    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"]
Jul 24 13:45:00.151277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.151261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"
Jul 24 13:45:00.160984 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.160962    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"
Jul 24 13:45:00.195706 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.195679    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"
Jul 24 13:45:00.316369 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.316342    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"]
Jul 24 13:45:00.319409 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:45:00.319376    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0101c1_3494_4157_9b9b_ad1d6d16650d.slice/crio-6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9 WatchSource:0}: Error finding container 6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9: Status 404 returned error can't find the container with id 6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9
Jul 24 13:45:00.341606 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.341580    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"]
Jul 24 13:45:00.344228 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:45:00.344204    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d491dbe_c7ec_4d59_8cb7_2777c8dbd5be.slice/crio-51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473 WatchSource:0}: Error finding container 51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473: Status 404 returned error can't find the container with id 51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473
Jul 24 13:45:00.362909 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.362845    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"]
Jul 24 13:45:00.365047 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:45:00.365026    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415813ab_d669_46bb_ade3_1f3cbf97a7ac.slice/crio-3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6 WatchSource:0}: Error finding container 3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6: Status 404 returned error can't find the container with id 3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6
Jul 24 13:45:00.611949 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.611895    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:45:00.615621 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.615601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.617970 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.617920    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b298\"/\"kube-root-ca.crt\""
Jul 24 13:45:00.618092 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.618026    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b298\"/\"default-dockercfg-2tv7r\""
Jul 24 13:45:00.618669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.618651    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b298\"/\"test-model-50b298-kserve-self-signed-certs\""
Jul 24 13:45:00.618781 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.618655    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b298\"/\"openshift-service-ca.crt\""
Jul 24 13:45:00.624870 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.624844    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:45:00.714721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.714721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.714721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.714721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714710    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.715084 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714794    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndgn\" (UniqueName: \"kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.715084 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.714892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.815587 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.815587 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.815851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.815851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.815851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndgn\" (UniqueName: \"kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.816032 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.815892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.816129 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.816108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.816181 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.816136    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.816230 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.816185    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.818019 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.817994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.818389 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.818369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.823834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.823810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndgn\" (UniqueName: \"kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn\") pod \"test-model-50b298-kserve-69948c4b57-zv5sr\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") " pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:00.927891 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:00.927846    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:01.073212 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.073185    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:45:01.074450 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:45:01.074423    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad85f8a9_3ac0_4240_8635_0da9a9b4855d.slice/crio-9e72f8e138a5b77259e4a816e9fab1fef8a148dc4bd9b8b25f20b63188719672 WatchSource:0}: Error finding container 9e72f8e138a5b77259e4a816e9fab1fef8a148dc4bd9b8b25f20b63188719672: Status 404 returned error can't find the container with id 9e72f8e138a5b77259e4a816e9fab1fef8a148dc4bd9b8b25f20b63188719672
Jul 24 13:45:01.245226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.245141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6" event={"ID":"dc0101c1-3494-4157-9b9b-ad1d6d16650d","Type":"ContainerStarted","Data":"76cbdf6b38c6c373439b1dca8792d7be36e7cb5889f998d4df791fad9a0cc14f"}
Jul 24 13:45:01.245643 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.245231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6" event={"ID":"dc0101c1-3494-4157-9b9b-ad1d6d16650d","Type":"ContainerStarted","Data":"6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9"}
Jul 24 13:45:01.246643 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.246614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl" event={"ID":"415813ab-d669-46bb-ade3-1f3cbf97a7ac","Type":"ContainerStarted","Data":"1fec58bb14c16d43c74cedac02edc63c7a2d347d83ca738c7c0100c5d68972c4"}
Jul 24 13:45:01.246754 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.246660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl" event={"ID":"415813ab-d669-46bb-ade3-1f3cbf97a7ac","Type":"ContainerStarted","Data":"3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6"}
Jul 24 13:45:01.248081 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.248060    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr" event={"ID":"4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be","Type":"ContainerStarted","Data":"dc8ef553100e37b187ffa92f95960a0ffb69b99bcfd7709be6ccd2fb7fac9259"}
Jul 24 13:45:01.248186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.248094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr" event={"ID":"4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be","Type":"ContainerStarted","Data":"51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473"}
Jul 24 13:45:01.249488 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.249466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" event={"ID":"ad85f8a9-3ac0-4240-8635-0da9a9b4855d","Type":"ContainerStarted","Data":"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"}
Jul 24 13:45:01.249559 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.249499    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" event={"ID":"ad85f8a9-3ac0-4240-8635-0da9a9b4855d","Type":"ContainerStarted","Data":"9e72f8e138a5b77259e4a816e9fab1fef8a148dc4bd9b8b25f20b63188719672"}
Jul 24 13:45:01.261628 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.261585    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6" podStartSLOduration=1.261569465 podStartE2EDuration="1.261569465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:01.260368914 +0000 UTC m=+2120.484852240" watchObservedRunningTime="2026-07-24 13:45:01.261569465 +0000 UTC m=+2120.486052793"
Jul 24 13:45:01.275527 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.275486    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr" podStartSLOduration=1.27547318 podStartE2EDuration="1.27547318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:01.273963172 +0000 UTC m=+2120.498446499" watchObservedRunningTime="2026-07-24 13:45:01.27547318 +0000 UTC m=+2120.499956506"
Jul 24 13:45:01.292547 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.292500    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" podStartSLOduration=1.292484599 podStartE2EDuration="1.292484599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:01.291375884 +0000 UTC m=+2120.515859222" watchObservedRunningTime="2026-07-24 13:45:01.292484599 +0000 UTC m=+2120.516967927"
Jul 24 13:45:01.305956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:01.305897    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl" podStartSLOduration=1.305884707 podStartE2EDuration="1.305884707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:01.30534544 +0000 UTC m=+2120.529828768" watchObservedRunningTime="2026-07-24 13:45:01.305884707 +0000 UTC m=+2120.530368034"
Jul 24 13:45:02.256062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.256026    2556 generic.go:356] "Generic (PLEG): container finished" podID="dc0101c1-3494-4157-9b9b-ad1d6d16650d" containerID="76cbdf6b38c6c373439b1dca8792d7be36e7cb5889f998d4df791fad9a0cc14f" exitCode=0
Jul 24 13:45:02.256477 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.256111    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6" event={"ID":"dc0101c1-3494-4157-9b9b-ad1d6d16650d","Type":"ContainerDied","Data":"76cbdf6b38c6c373439b1dca8792d7be36e7cb5889f998d4df791fad9a0cc14f"}
Jul 24 13:45:02.257538 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.257514    2556 generic.go:356] "Generic (PLEG): container finished" podID="415813ab-d669-46bb-ade3-1f3cbf97a7ac" containerID="1fec58bb14c16d43c74cedac02edc63c7a2d347d83ca738c7c0100c5d68972c4" exitCode=0
Jul 24 13:45:02.257686 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.257557    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl" event={"ID":"415813ab-d669-46bb-ade3-1f3cbf97a7ac","Type":"ContainerDied","Data":"1fec58bb14c16d43c74cedac02edc63c7a2d347d83ca738c7c0100c5d68972c4"}
Jul 24 13:45:02.258985 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.258965    2556 generic.go:356] "Generic (PLEG): container finished" podID="4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be" containerID="dc8ef553100e37b187ffa92f95960a0ffb69b99bcfd7709be6ccd2fb7fac9259" exitCode=0
Jul 24 13:45:02.259117 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:02.259038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr" event={"ID":"4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be","Type":"ContainerDied","Data":"dc8ef553100e37b187ffa92f95960a0ffb69b99bcfd7709be6ccd2fb7fac9259"}
Jul 24 13:45:03.428543 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:03.428518    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"
Jul 24 13:45:03.432065 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:03.432044    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"
Jul 24 13:45:03.435745 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:03.435719    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"
Jul 24 13:45:04.276694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.276662    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6"
Jul 24 13:45:04.276694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.276670    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748345-gmdw6" event={"ID":"dc0101c1-3494-4157-9b9b-ad1d6d16650d","Type":"ContainerDied","Data":"6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9"}
Jul 24 13:45:04.276921 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.276751    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e264f48fa4ae9850c8cbbfd2af84c81cb26b317295ae7c21e6f6edf331a1df9"
Jul 24 13:45:04.278109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.278086    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl" event={"ID":"415813ab-d669-46bb-ade3-1f3cbf97a7ac","Type":"ContainerDied","Data":"3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6"}
Jul 24 13:45:04.278237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.278121    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c365ac01222f386d497b8b741439b315474db4584242f954c2fd38f4987e9e6"
Jul 24 13:45:04.278237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.278096    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"
Jul 24 13:45:04.279375 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.279356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr" event={"ID":"4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be","Type":"ContainerDied","Data":"51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473"}
Jul 24 13:45:04.279375 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.279371    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"
Jul 24 13:45:04.279505 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:04.279388    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51768dbf0cbc47e3cb77156943d001b8065f3a2d7e52a39272971f638c626473"
Jul 24 13:45:10.929034 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:10.928998    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:10.929034 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:10.929041    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:10.941689 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:10.941653    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:11.327620 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:11.327590    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:45:23.616969 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.616916    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:45:23.617359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617317    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be" containerName="cleanup"
Jul 24 13:45:23.617359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617329    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be" containerName="cleanup"
Jul 24 13:45:23.617359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617340    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0101c1-3494-4157-9b9b-ad1d6d16650d" containerName="cleanup"
Jul 24 13:45:23.617359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617345    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0101c1-3494-4157-9b9b-ad1d6d16650d" containerName="cleanup"
Jul 24 13:45:23.617359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617359    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415813ab-d669-46bb-ade3-1f3cbf97a7ac" containerName="cleanup"
Jul 24 13:45:23.617529 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617364    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="415813ab-d669-46bb-ade3-1f3cbf97a7ac" containerName="cleanup"
Jul 24 13:45:23.617529 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617434    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="415813ab-d669-46bb-ade3-1f3cbf97a7ac" containerName="cleanup"
Jul 24 13:45:23.617529 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617441    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be" containerName="cleanup"
Jul 24 13:45:23.617529 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.617446    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0101c1-3494-4157-9b9b-ad1d6d16650d" containerName="cleanup"
Jul 24 13:45:23.620579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.620564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.622997 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.622977    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7918ee\"/\"kube-root-ca.crt\""
Jul 24 13:45:23.624010 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.623993    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7918ee\"/\"openshift-service-ca.crt\""
Jul 24 13:45:23.624124 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.624102    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7918ee\"/\"test-model-7918ee-kserve-self-signed-certs\""
Jul 24 13:45:23.624175 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.624104    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7918ee\"/\"default-dockercfg-gstlr\""
Jul 24 13:45:23.632952 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.632909    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:45:23.727830 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.727797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.728039 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.727838    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.728039 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.727885    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.728039 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.727976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.728039 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.727999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkldt\" (UniqueName: \"kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.728253 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.728090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829266 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829321    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829445 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkldt\" (UniqueName: \"kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829667 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829769 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.829888 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.829866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.831754 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.831735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.831862 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.831845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.837612 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.837588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkldt\" (UniqueName: \"kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt\") pod \"test-model-7918ee-kserve-764967ffff-899rd\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") " pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:23.929508 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:23.929412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:24.062721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:24.062694    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:45:24.064594 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:45:24.064561    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1cdd541_0937_402b_85cb_dfbe77840b05.slice/crio-f6ee26df7f1dcfdf9042466989e192cc804f13eb837a4f5faa3e296e0718058f WatchSource:0}: Error finding container f6ee26df7f1dcfdf9042466989e192cc804f13eb837a4f5faa3e296e0718058f: Status 404 returned error can't find the container with id f6ee26df7f1dcfdf9042466989e192cc804f13eb837a4f5faa3e296e0718058f
Jul 24 13:45:24.382503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:24.382467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" event={"ID":"a1cdd541-0937-402b-85cb-dfbe77840b05","Type":"ContainerStarted","Data":"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"}
Jul 24 13:45:24.382503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:24.382510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" event={"ID":"a1cdd541-0937-402b-85cb-dfbe77840b05","Type":"ContainerStarted","Data":"f6ee26df7f1dcfdf9042466989e192cc804f13eb837a4f5faa3e296e0718058f"}
Jul 24 13:45:24.402464 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:24.402406    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" podStartSLOduration=1.402390913 podStartE2EDuration="1.402390913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:24.400987108 +0000 UTC m=+2143.625470432" watchObservedRunningTime="2026-07-24 13:45:24.402390913 +0000 UTC m=+2143.626874244"
Jul 24 13:45:33.930261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:33.930156    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:33.930261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:33.930243    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:33.943139 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:33.943109    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:45:34.444793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:45:34.444761    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:46:11.747612 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:11.747580    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:46:12.708665 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:12.708631    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:46:23.270085 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.270041    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:46:23.270670 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.270288    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" podUID="2d748d26-9024-4168-944d-a0d7db283141" containerName="maas-api" containerID="cri-o://9bebc18fc3847c1236ed3c75ac42e7ba1df83bccc1b3e4b2d4978c9b05731283" gracePeriod=30
Jul 24 13:46:23.308506 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.308465    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"]
Jul 24 13:46:23.310965 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.310926    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b298-29748345-qmspl"]
Jul 24 13:46:23.342397 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.342362    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415813ab-d669-46bb-ade3-1f3cbf97a7ac" path="/var/lib/kubelet/pods/415813ab-d669-46bb-ade3-1f3cbf97a7ac/volumes"
Jul 24 13:46:23.682833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.682802    2556 generic.go:356] "Generic (PLEG): container finished" podID="2d748d26-9024-4168-944d-a0d7db283141" containerID="9bebc18fc3847c1236ed3c75ac42e7ba1df83bccc1b3e4b2d4978c9b05731283" exitCode=0
Jul 24 13:46:23.683037 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.682869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" event={"ID":"2d748d26-9024-4168-944d-a0d7db283141","Type":"ContainerDied","Data":"9bebc18fc3847c1236ed3c75ac42e7ba1df83bccc1b3e4b2d4978c9b05731283"}
Jul 24 13:46:23.718912 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.718890    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:46:23.801414 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.801316    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkjd7\" (UniqueName: \"kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7\") pod \"2d748d26-9024-4168-944d-a0d7db283141\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") "
Jul 24 13:46:23.801414 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.801415    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca\") pod \"2d748d26-9024-4168-944d-a0d7db283141\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") "
Jul 24 13:46:23.801652 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.801434    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") pod \"2d748d26-9024-4168-944d-a0d7db283141\" (UID: \"2d748d26-9024-4168-944d-a0d7db283141\") "
Jul 24 13:46:23.801735 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.801713    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d748d26-9024-4168-944d-a0d7db283141" (UID: "2d748d26-9024-4168-944d-a0d7db283141"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:46:23.803723 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.803699    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7" (OuterVolumeSpecName: "kube-api-access-tkjd7") pod "2d748d26-9024-4168-944d-a0d7db283141" (UID: "2d748d26-9024-4168-944d-a0d7db283141"). InnerVolumeSpecName "kube-api-access-tkjd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:23.803826 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.803774    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d748d26-9024-4168-944d-a0d7db283141" (UID: "2d748d26-9024-4168-944d-a0d7db283141"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:46:23.902157 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.902120    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d748d26-9024-4168-944d-a0d7db283141-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:23.902157 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.902150    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d748d26-9024-4168-944d-a0d7db283141-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:23.902157 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:23.902160    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjd7\" (UniqueName: \"kubernetes.io/projected/2d748d26-9024-4168-944d-a0d7db283141-kube-api-access-tkjd7\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:24.689550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:24.689516    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"
Jul 24 13:46:24.689550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:24.689532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb" event={"ID":"2d748d26-9024-4168-944d-a0d7db283141","Type":"ContainerDied","Data":"d1c35f229202f367d17aa0bd4c116a23662dcc92bb1636b1e40d91ab16892e38"}
Jul 24 13:46:24.690126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:24.689596    2556 scope.go:117] "RemoveContainer" containerID="9bebc18fc3847c1236ed3c75ac42e7ba1df83bccc1b3e4b2d4978c9b05731283"
Jul 24 13:46:24.712248 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:24.712216    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:46:24.716724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:24.716700    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b298-7c85b8475c-ztltb"]
Jul 24 13:46:25.340889 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:25.340851    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d748d26-9024-4168-944d-a0d7db283141" path="/var/lib/kubelet/pods/2d748d26-9024-4168-944d-a0d7db283141/volumes"
Jul 24 13:46:26.766988 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:26.766919    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" podUID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" containerName="main" containerID="cri-o://e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6" gracePeriod=45
Jul 24 13:46:27.021955 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.021883    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:46:27.029605 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029585    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029725 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029640    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029725 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029661    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029725 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029694    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bndgn\" (UniqueName: \"kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029725 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir\") pod \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\" (UID: \"ad85f8a9-3ac0-4240-8635-0da9a9b4855d\") "
Jul 24 13:46:27.029954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029876    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:27.030076 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.029988    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home" (OuterVolumeSpecName: "home") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:27.030131 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.030120    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.030176 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.030138    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.030176 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.030139    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:27.031970 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.031917    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm" (OuterVolumeSpecName: "dshm") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:27.032071 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.031985    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:46:27.032071 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.032003    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn" (OuterVolumeSpecName: "kube-api-access-bndgn") pod "ad85f8a9-3ac0-4240-8635-0da9a9b4855d" (UID: "ad85f8a9-3ac0-4240-8635-0da9a9b4855d"). InnerVolumeSpecName "kube-api-access-bndgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:27.130846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.130808    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndgn\" (UniqueName: \"kubernetes.io/projected/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-kube-api-access-bndgn\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.130846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.130838    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.130846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.130849    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.130846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.130858    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad85f8a9-3ac0-4240-8635-0da9a9b4855d-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:27.714834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.714732    2556 generic.go:356] "Generic (PLEG): container finished" podID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" containerID="e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6" exitCode=0
Jul 24 13:46:27.714834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.714777    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"
Jul 24 13:46:27.714834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.714793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" event={"ID":"ad85f8a9-3ac0-4240-8635-0da9a9b4855d","Type":"ContainerDied","Data":"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"}
Jul 24 13:46:27.715135 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.714835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr" event={"ID":"ad85f8a9-3ac0-4240-8635-0da9a9b4855d","Type":"ContainerDied","Data":"9e72f8e138a5b77259e4a816e9fab1fef8a148dc4bd9b8b25f20b63188719672"}
Jul 24 13:46:27.715135 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.714857    2556 scope.go:117] "RemoveContainer" containerID="e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"
Jul 24 13:46:27.723551 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.723531    2556 scope.go:117] "RemoveContainer" containerID="e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"
Jul 24 13:46:27.723804 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:46:27.723785    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6\": container with ID starting with e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6 not found: ID does not exist" containerID="e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"
Jul 24 13:46:27.723854 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.723812    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6"} err="failed to get container status \"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6\": rpc error: code = NotFound desc = could not find container \"e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6\": container with ID starting with e572886ee9771be5486749292928989c32df18447ebdb6e56b31217e87e5d6b6 not found: ID does not exist"
Jul 24 13:46:27.727953 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.727882    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" podUID="a1cdd541-0937-402b-85cb-dfbe77840b05" containerName="main" containerID="cri-o://db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49" gracePeriod=45
Jul 24 13:46:27.732956 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.732914    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:46:27.738613 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.738588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50b298/test-model-50b298-kserve-69948c4b57-zv5sr"]
Jul 24 13:46:27.981167 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:27.981142    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:46:28.039840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.039804    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.039853    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.039890    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkldt\" (UniqueName: \"kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.039922    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.039953    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040019    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home\") pod \"a1cdd541-0937-402b-85cb-dfbe77840b05\" (UID: \"a1cdd541-0937-402b-85cb-dfbe77840b05\") "
Jul 24 13:46:28.040259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040129    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:28.040259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040191    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:28.040370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040328    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.040370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040361    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.040449 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.040364    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home" (OuterVolumeSpecName: "home") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:28.042419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.042394    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt" (OuterVolumeSpecName: "kube-api-access-zkldt") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "kube-api-access-zkldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:28.042419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.042394    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm" (OuterVolumeSpecName: "dshm") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:46:28.042588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.042393    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1cdd541-0937-402b-85cb-dfbe77840b05" (UID: "a1cdd541-0937-402b-85cb-dfbe77840b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:46:28.141384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.141348    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.141384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.141380    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkldt\" (UniqueName: \"kubernetes.io/projected/a1cdd541-0937-402b-85cb-dfbe77840b05-kube-api-access-zkldt\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.141384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.141390    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cdd541-0937-402b-85cb-dfbe77840b05-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.141620 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.141400    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cdd541-0937-402b-85cb-dfbe77840b05-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:28.723563 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.723527    2556 generic.go:356] "Generic (PLEG): container finished" podID="a1cdd541-0937-402b-85cb-dfbe77840b05" containerID="db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49" exitCode=0
Jul 24 13:46:28.723764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.723575    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"
Jul 24 13:46:28.723764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.723580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" event={"ID":"a1cdd541-0937-402b-85cb-dfbe77840b05","Type":"ContainerDied","Data":"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"}
Jul 24 13:46:28.723764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.723619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd" event={"ID":"a1cdd541-0937-402b-85cb-dfbe77840b05","Type":"ContainerDied","Data":"f6ee26df7f1dcfdf9042466989e192cc804f13eb837a4f5faa3e296e0718058f"}
Jul 24 13:46:28.723764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.723641    2556 scope.go:117] "RemoveContainer" containerID="db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"
Jul 24 13:46:28.732851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.732834    2556 scope.go:117] "RemoveContainer" containerID="db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"
Jul 24 13:46:28.733156 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:46:28.733134    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49\": container with ID starting with db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49 not found: ID does not exist" containerID="db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"
Jul 24 13:46:28.733215 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.733165    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49"} err="failed to get container status \"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49\": rpc error: code = NotFound desc = could not find container \"db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49\": container with ID starting with db2814b1537dfc444f64cc25c5e1ee4665bc465c79718646ca6557e956b3fe49 not found: ID does not exist"
Jul 24 13:46:28.744233 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.744201    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:46:28.748195 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:28.748168    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7918ee/test-model-7918ee-kserve-764967ffff-899rd"]
Jul 24 13:46:29.341359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:29.341322    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1cdd541-0937-402b-85cb-dfbe77840b05" path="/var/lib/kubelet/pods/a1cdd541-0937-402b-85cb-dfbe77840b05/volumes"
Jul 24 13:46:29.341732 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:29.341693    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" path="/var/lib/kubelet/pods/ad85f8a9-3ac0-4240-8635-0da9a9b4855d/volumes"
Jul 24 13:46:34.766278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766246    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"]
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766624    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" containerName="main"
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766636    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" containerName="main"
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766645    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1cdd541-0937-402b-85cb-dfbe77840b05" containerName="main"
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766651    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1cdd541-0937-402b-85cb-dfbe77840b05" containerName="main"
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766676    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d748d26-9024-4168-944d-a0d7db283141" containerName="maas-api"
Jul 24 13:46:34.766683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766681    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d748d26-9024-4168-944d-a0d7db283141" containerName="maas-api"
Jul 24 13:46:34.766868 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766740    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d748d26-9024-4168-944d-a0d7db283141" containerName="maas-api"
Jul 24 13:46:34.766868 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766747    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1cdd541-0937-402b-85cb-dfbe77840b05" containerName="main"
Jul 24 13:46:34.766868 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.766756    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad85f8a9-3ac0-4240-8635-0da9a9b4855d" containerName="main"
Jul 24 13:46:34.771227 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.771208    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:34.773738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.773713    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:46:34.777540 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.777512    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"]
Jul 24 13:46:34.803357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.803315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn\" (UID: \"7f860bb3-af0e-4a75-8356-e45e9dcab7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:34.904272 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.904229    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn\" (UID: \"7f860bb3-af0e-4a75-8356-e45e9dcab7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:34.904870 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:34.904849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn\" (UID: \"7f860bb3-af0e-4a75-8356-e45e9dcab7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:35.083304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:35.083257    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:35.220531 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:35.220501    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"]
Jul 24 13:46:35.222892 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:46:35.222867    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f860bb3_af0e_4a75_8356_e45e9dcab7d9.slice/crio-147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5 WatchSource:0}: Error finding container 147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5: Status 404 returned error can't find the container with id 147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5
Jul 24 13:46:35.764043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:35.763926    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn" event={"ID":"7f860bb3-af0e-4a75-8356-e45e9dcab7d9","Type":"ContainerStarted","Data":"0cacbc9633fe59f895ef547e1aada149a5905fbe25fbbb60a5c433d9e2ca2b62"}
Jul 24 13:46:35.764043 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:35.764005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn" event={"ID":"7f860bb3-af0e-4a75-8356-e45e9dcab7d9","Type":"ContainerStarted","Data":"147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5"}
Jul 24 13:46:35.788395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:35.788326    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn" podStartSLOduration=1.788309064 podStartE2EDuration="1.788309064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:35.786503196 +0000 UTC m=+2215.010986524" watchObservedRunningTime="2026-07-24 13:46:35.788309064 +0000 UTC m=+2215.012792390"
Jul 24 13:46:36.771395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:36.771359    2556 generic.go:356] "Generic (PLEG): container finished" podID="7f860bb3-af0e-4a75-8356-e45e9dcab7d9" containerID="0cacbc9633fe59f895ef547e1aada149a5905fbe25fbbb60a5c433d9e2ca2b62" exitCode=0
Jul 24 13:46:36.771581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:36.771446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn" event={"ID":"7f860bb3-af0e-4a75-8356-e45e9dcab7d9","Type":"ContainerDied","Data":"0cacbc9633fe59f895ef547e1aada149a5905fbe25fbbb60a5c433d9e2ca2b62"}
Jul 24 13:46:37.918701 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:37.918661    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:38.032795 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.032694    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca\") pod \"7f860bb3-af0e-4a75-8356-e45e9dcab7d9\" (UID: \"7f860bb3-af0e-4a75-8356-e45e9dcab7d9\") "
Jul 24 13:46:38.033148 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.033127    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f860bb3-af0e-4a75-8356-e45e9dcab7d9" (UID: "7f860bb3-af0e-4a75-8356-e45e9dcab7d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:46:38.133492 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.133455    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f860bb3-af0e-4a75-8356-e45e9dcab7d9-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:38.784094 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.784061    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"
Jul 24 13:46:38.784094 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.784087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn" event={"ID":"7f860bb3-af0e-4a75-8356-e45e9dcab7d9","Type":"ContainerDied","Data":"147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5"}
Jul 24 13:46:38.784315 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:38.784134    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147764b718d7c8c9e2de516651bc1d65712322fa84f2f79ccc7e8503a3c288a5"
Jul 24 13:46:41.962091 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:41.962057    2556 scope.go:117] "RemoveContainer" containerID="53461f874c99056d3e34652ef1af6c4e6dff26428c662544c23630ba2024a25f"
Jul 24 13:46:45.240214 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.240178    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:46:45.240677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.240482    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" podUID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" containerName="maas-api" containerID="cri-o://ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1" gracePeriod=30
Jul 24 13:46:45.256814 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.256785    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"]
Jul 24 13:46:45.262207 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.262179    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7918ee-29748345-6hdfr"]
Jul 24 13:46:45.341270 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.341239    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be" path="/var/lib/kubelet/pods/4d491dbe-c7ec-4d59-8cb7-2777c8dbd5be/volumes"
Jul 24 13:46:45.341484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.341471    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:46:45.341664 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.341647    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" podUID="31838071-57f0-4185-9241-1e84152ae76c" containerName="payload-pre-processing" containerID="cri-o://2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939" gracePeriod=30
Jul 24 13:46:45.502284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.502208    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:46:45.590328 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.590295    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:46:45.607366 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.607314    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") pod \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") "
Jul 24 13:46:45.607569 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.607384    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnc5n\" (UniqueName: \"kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n\") pod \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") "
Jul 24 13:46:45.607569 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.607461    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca\") pod \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\" (UID: \"a7695ef7-dd58-4329-b6e1-712053fbc1f1\") "
Jul 24 13:46:45.607915 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.607867    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7695ef7-dd58-4329-b6e1-712053fbc1f1" (UID: "a7695ef7-dd58-4329-b6e1-712053fbc1f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:46:45.610360 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.610307    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n" (OuterVolumeSpecName: "kube-api-access-xnc5n") pod "a7695ef7-dd58-4329-b6e1-712053fbc1f1" (UID: "a7695ef7-dd58-4329-b6e1-712053fbc1f1"). InnerVolumeSpecName "kube-api-access-xnc5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:45.610484 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.610467    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7695ef7-dd58-4329-b6e1-712053fbc1f1" (UID: "a7695ef7-dd58-4329-b6e1-712053fbc1f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:46:45.708866 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.708833    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume\") pod \"31838071-57f0-4185-9241-1e84152ae76c\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") "
Jul 24 13:46:45.709072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.708972    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8p2m\" (UniqueName: \"kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m\") pod \"31838071-57f0-4185-9241-1e84152ae76c\" (UID: \"31838071-57f0-4185-9241-1e84152ae76c\") "
Jul 24 13:46:45.709305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.709281    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31838071-57f0-4185-9241-1e84152ae76c" (UID: "31838071-57f0-4185-9241-1e84152ae76c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:46:45.709391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.709290    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7695ef7-dd58-4329-b6e1-712053fbc1f1-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:45.709391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.709331    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnc5n\" (UniqueName: \"kubernetes.io/projected/a7695ef7-dd58-4329-b6e1-712053fbc1f1-kube-api-access-xnc5n\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:45.709391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.709353    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7695ef7-dd58-4329-b6e1-712053fbc1f1-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:45.711243 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.711218    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m" (OuterVolumeSpecName: "kube-api-access-d8p2m") pod "31838071-57f0-4185-9241-1e84152ae76c" (UID: "31838071-57f0-4185-9241-1e84152ae76c"). InnerVolumeSpecName "kube-api-access-d8p2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:45.810325 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.810286    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31838071-57f0-4185-9241-1e84152ae76c-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:45.810325 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.810314    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p2m\" (UniqueName: \"kubernetes.io/projected/31838071-57f0-4185-9241-1e84152ae76c-kube-api-access-d8p2m\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:46:45.822631 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.822595    2556 generic.go:356] "Generic (PLEG): container finished" podID="31838071-57f0-4185-9241-1e84152ae76c" containerID="2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939" exitCode=0
Jul 24 13:46:45.822840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.822644    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"
Jul 24 13:46:45.822840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.822688    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" event={"ID":"31838071-57f0-4185-9241-1e84152ae76c","Type":"ContainerDied","Data":"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"}
Jul 24 13:46:45.822840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.822745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9" event={"ID":"31838071-57f0-4185-9241-1e84152ae76c","Type":"ContainerDied","Data":"7929fcd83cb70c2406e02b51898cf3d6329eb9a75284986b72a9df52f5ab7c02"}
Jul 24 13:46:45.822840 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.822771    2556 scope.go:117] "RemoveContainer" containerID="2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"
Jul 24 13:46:45.824115 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.824093    2556 generic.go:356] "Generic (PLEG): container finished" podID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" containerID="ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1" exitCode=0
Jul 24 13:46:45.824258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.824138    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"
Jul 24 13:46:45.824258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.824159    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" event={"ID":"a7695ef7-dd58-4329-b6e1-712053fbc1f1","Type":"ContainerDied","Data":"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"}
Jul 24 13:46:45.824258 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.824197    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq" event={"ID":"a7695ef7-dd58-4329-b6e1-712053fbc1f1","Type":"ContainerDied","Data":"f568f394d6d48c7d1108dcf64bc15a861f2bcf3f3116aad3e6a72192bf8d1ffe"}
Jul 24 13:46:45.833878 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.833857    2556 scope.go:117] "RemoveContainer" containerID="2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"
Jul 24 13:46:45.834527 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:46:45.834502    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939\": container with ID starting with 2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939 not found: ID does not exist" containerID="2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"
Jul 24 13:46:45.834633 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.834535    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939"} err="failed to get container status \"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939\": rpc error: code = NotFound desc = could not find container \"2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939\": container with ID starting with 2c62538a902febb713783a7015227cb2ab8878902a11b9cb75e524d959e7f939 not found: ID does not exist"
Jul 24 13:46:45.834633 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.834554    2556 scope.go:117] "RemoveContainer" containerID="ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"
Jul 24 13:46:45.843702 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.843681    2556 scope.go:117] "RemoveContainer" containerID="ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"
Jul 24 13:46:45.844033 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:46:45.844006    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1\": container with ID starting with ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1 not found: ID does not exist" containerID="ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"
Jul 24 13:46:45.844118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.844045    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1"} err="failed to get container status \"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1\": rpc error: code = NotFound desc = could not find container \"ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1\": container with ID starting with ae5d1b0cee98125cc855bc6c6e8ee2eed8024e973a41c87d38d7acb647e1fbb1 not found: ID does not exist"
Jul 24 13:46:45.849518 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.849489    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:46:45.855730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.855697    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7918ee-7799b86f57-w4wbq"]
Jul 24 13:46:45.865824 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.865786    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:46:45.873450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:45.873416    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7918ee-756c4dfb6d-2zdf9"]
Jul 24 13:46:47.344641 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:47.344605    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31838071-57f0-4185-9241-1e84152ae76c" path="/var/lib/kubelet/pods/31838071-57f0-4185-9241-1e84152ae76c/volumes"
Jul 24 13:46:47.345149 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:47.344997    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" path="/var/lib/kubelet/pods/a7695ef7-dd58-4329-b6e1-712053fbc1f1/volumes"
Jul 24 13:46:49.841697 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:49.841660    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"]
Jul 24 13:46:49.846342 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:49.846312    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7918ee-p5nkn"]
Jul 24 13:46:51.342352 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:46:51.342318    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f860bb3-af0e-4a75-8356-e45e9dcab7d9" path="/var/lib/kubelet/pods/7f860bb3-af0e-4a75-8356-e45e9dcab7d9/volumes"
Jul 24 13:47:00.135211 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135170    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135681    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31838071-57f0-4185-9241-1e84152ae76c" containerName="payload-pre-processing"
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135701    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="31838071-57f0-4185-9241-1e84152ae76c" containerName="payload-pre-processing"
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135724    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" containerName="maas-api"
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135734    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" containerName="maas-api"
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135743    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f860bb3-af0e-4a75-8356-e45e9dcab7d9" containerName="revoke-keys"
Jul 24 13:47:00.135793 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135752    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f860bb3-af0e-4a75-8356-e45e9dcab7d9" containerName="revoke-keys"
Jul 24 13:47:00.136126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135848    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f860bb3-af0e-4a75-8356-e45e9dcab7d9" containerName="revoke-keys"
Jul 24 13:47:00.136126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135865    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="31838071-57f0-4185-9241-1e84152ae76c" containerName="payload-pre-processing"
Jul 24 13:47:00.136126 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.135875    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7695ef7-dd58-4329-b6e1-712053fbc1f1" containerName="maas-api"
Jul 24 13:47:00.140393 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.140372    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.142678 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.142652    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-502742-serving-cert\""
Jul 24 13:47:00.142813 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.142720    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:47:00.151898 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.151871    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:00.242129 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.242080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.242129 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.242137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.242370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.242190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crp56\" (UniqueName: \"kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.343512 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.343475    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.343727 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.343521    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.343727 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.343570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crp56\" (UniqueName: \"kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.343727 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:47:00.343708    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-502742-serving-cert: secret "maas-api-e2e-api-a-502742-serving-cert" not found
Jul 24 13:47:00.343969 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:47:00.343798    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls podName:be2262be-d0a5-4803-81d6-07a06b46e90e nodeName:}" failed. No retries permitted until 2026-07-24 13:47:00.843776215 +0000 UTC m=+2240.068259521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls") pod "maas-api-e2e-api-a-502742-9d6b6696-2jpz6" (UID: "be2262be-d0a5-4803-81d6-07a06b46e90e") : secret "maas-api-e2e-api-a-502742-serving-cert" not found
Jul 24 13:47:00.344171 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.344151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.376099 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.376063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crp56\" (UniqueName: \"kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.848269 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.848225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:00.850837 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:00.850809    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") pod \"maas-api-e2e-api-a-502742-9d6b6696-2jpz6\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:01.050819 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.050770    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:01.115162 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.115124    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:01.119912 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.119883    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.122527 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.122504    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-502742\""
Jul 24 13:47:01.122683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.122524    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-502742-dockercfg-2d6s2\""
Jul 24 13:47:01.127503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.127478    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:01.197833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.197807    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:01.201355 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:47:01.201325    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2262be_d0a5_4803_81d6_07a06b46e90e.slice/crio-efb086ea891fac71941850362e8ac7e573f8b21dad2ee6f65f750e6a71d4664e WatchSource:0}: Error finding container efb086ea891fac71941850362e8ac7e573f8b21dad2ee6f65f750e6a71d4664e: Status 404 returned error can't find the container with id efb086ea891fac71941850362e8ac7e573f8b21dad2ee6f65f750e6a71d4664e
Jul 24 13:47:01.202662 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.202644    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:47:01.251948 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.251908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.252128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.251972    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbh9x\" (UniqueName: \"kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.352499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.352465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.352499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.352504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh9x\" (UniqueName: \"kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.353316 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.353269    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.361106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.361083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbh9x\" (UniqueName: \"kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x\") pod \"payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.433639 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.433597    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.574646 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.574503    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:01.576765 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:47:01.576741    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bac958_6515_4871_b031_96c461f4537b.slice/crio-a63966fbe8c62a07915d03de03b1dacc418dbb3a18cdccce177af3c55beff915 WatchSource:0}: Error finding container a63966fbe8c62a07915d03de03b1dacc418dbb3a18cdccce177af3c55beff915: Status 404 returned error can't find the container with id a63966fbe8c62a07915d03de03b1dacc418dbb3a18cdccce177af3c55beff915
Jul 24 13:47:01.905724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.905622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" event={"ID":"94bac958-6515-4871-b031-96c461f4537b","Type":"ContainerStarted","Data":"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"}
Jul 24 13:47:01.905724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.905683    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" event={"ID":"94bac958-6515-4871-b031-96c461f4537b","Type":"ContainerStarted","Data":"a63966fbe8c62a07915d03de03b1dacc418dbb3a18cdccce177af3c55beff915"}
Jul 24 13:47:01.905963 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.905724    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:01.907118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.907091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" event={"ID":"be2262be-d0a5-4803-81d6-07a06b46e90e","Type":"ContainerStarted","Data":"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"}
Jul 24 13:47:01.907255 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.907140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" event={"ID":"be2262be-d0a5-4803-81d6-07a06b46e90e","Type":"ContainerStarted","Data":"efb086ea891fac71941850362e8ac7e573f8b21dad2ee6f65f750e6a71d4664e"}
Jul 24 13:47:01.907255 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.907205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:01.922621 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.922558    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" podStartSLOduration=0.9225356 podStartE2EDuration="922.5356ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:01.920266376 +0000 UTC m=+2241.144749705" watchObservedRunningTime="2026-07-24 13:47:01.9225356 +0000 UTC m=+2241.147018927"
Jul 24 13:47:01.946065 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:01.946000    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" podStartSLOduration=1.4659081 podStartE2EDuration="1.945983251s" totalImagesPullingTime="480.075151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:01.9442006 +0000 UTC m=+2241.168683926" watchObservedRunningTime="2026-07-24 13:47:01.945983251 +0000 UTC m=+2241.170466605"
Jul 24 13:47:07.917452 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:07.917420    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:08.174298 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.174208    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:47:08.177588 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.177566    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.182534 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.182515    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-799cec-serving-cert\""
Jul 24 13:47:08.192053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.192031    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:47:08.314655 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.314617    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.314655 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.314665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g6w\" (UniqueName: \"kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.314897 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.314736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.415457 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.415414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.415457 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.415466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g6w\" (UniqueName: \"kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.415747 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.415496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.416178 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.416149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.418121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.418102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.426293 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.426236    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g6w\" (UniqueName: \"kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w\") pod \"maas-api-e2e-api-b-799cec-5c4998c94c-b9n95\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.487633 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.487592    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:08.623081 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.623052    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:47:08.625488 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:47:08.625461    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e511262_3727_489a_bb1c_d829661b3716.slice/crio-8130b74b61405d7746609a7cd0d668da6d0ed4dc4565830f243cab9b175596d3 WatchSource:0}: Error finding container 8130b74b61405d7746609a7cd0d668da6d0ed4dc4565830f243cab9b175596d3: Status 404 returned error can't find the container with id 8130b74b61405d7746609a7cd0d668da6d0ed4dc4565830f243cab9b175596d3
Jul 24 13:47:08.940729 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:08.940690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" event={"ID":"8e511262-3727-489a-bb1c-d829661b3716","Type":"ContainerStarted","Data":"8130b74b61405d7746609a7cd0d668da6d0ed4dc4565830f243cab9b175596d3"}
Jul 24 13:47:09.946740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:09.946703    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" event={"ID":"8e511262-3727-489a-bb1c-d829661b3716","Type":"ContainerStarted","Data":"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"}
Jul 24 13:47:09.947186 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:09.946791    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:09.963103 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:09.963051    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" podStartSLOduration=1.563800274 podStartE2EDuration="1.963034429s" totalImagesPullingTime="399.234155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:09.961863334 +0000 UTC m=+2249.186346662" watchObservedRunningTime="2026-07-24 13:47:09.963034429 +0000 UTC m=+2249.187517755"
Jul 24 13:47:12.913297 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:12.913266    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:15.956889 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:15.956862    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:47:20.861291 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:20.861252    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"]
Jul 24 13:47:20.864587 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:20.864570    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:20.866625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:20.866602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:47:20.872217 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:20.872191    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"]
Jul 24 13:47:20.939607 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:20.939576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-502742-2jt44\" (UID: \"2a80fd3e-aa35-43ea-82a3-54752beddf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:21.040375 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:21.040328    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-502742-2jt44\" (UID: \"2a80fd3e-aa35-43ea-82a3-54752beddf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:21.040984 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:21.040967    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-502742-2jt44\" (UID: \"2a80fd3e-aa35-43ea-82a3-54752beddf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:21.176775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:21.176683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:21.316291 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:21.316267    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"]
Jul 24 13:47:21.318450 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:47:21.318415    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a80fd3e_aa35_43ea_82a3_54752beddf4b.slice/crio-7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848 WatchSource:0}: Error finding container 7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848: Status 404 returned error can't find the container with id 7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848
Jul 24 13:47:22.010651 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:22.010557    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44" event={"ID":"2a80fd3e-aa35-43ea-82a3-54752beddf4b","Type":"ContainerStarted","Data":"3b0cec819e77c26412c555084fb09ef3c6b7142e60238f917a9e65ef5c79b2b8"}
Jul 24 13:47:22.010651 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:22.010642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44" event={"ID":"2a80fd3e-aa35-43ea-82a3-54752beddf4b","Type":"ContainerStarted","Data":"7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848"}
Jul 24 13:47:22.028846 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:22.028794    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44" podStartSLOduration=2.028778186 podStartE2EDuration="2.028778186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:22.027523525 +0000 UTC m=+2261.252006853" watchObservedRunningTime="2026-07-24 13:47:22.028778186 +0000 UTC m=+2261.253261513"
Jul 24 13:47:23.017440 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:23.017405    2556 generic.go:356] "Generic (PLEG): container finished" podID="2a80fd3e-aa35-43ea-82a3-54752beddf4b" containerID="3b0cec819e77c26412c555084fb09ef3c6b7142e60238f917a9e65ef5c79b2b8" exitCode=0
Jul 24 13:47:23.017860 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:23.017491    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44" event={"ID":"2a80fd3e-aa35-43ea-82a3-54752beddf4b","Type":"ContainerDied","Data":"3b0cec819e77c26412c555084fb09ef3c6b7142e60238f917a9e65ef5c79b2b8"}
Jul 24 13:47:24.162387 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:24.162362    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:24.271465 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:24.271427    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca\") pod \"2a80fd3e-aa35-43ea-82a3-54752beddf4b\" (UID: \"2a80fd3e-aa35-43ea-82a3-54752beddf4b\") "
Jul 24 13:47:24.271835 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:24.271806    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a80fd3e-aa35-43ea-82a3-54752beddf4b" (UID: "2a80fd3e-aa35-43ea-82a3-54752beddf4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:47:24.372209 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:24.372098    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a80fd3e-aa35-43ea-82a3-54752beddf4b-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:25.029120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:25.029089    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"
Jul 24 13:47:25.029275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:25.029091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44" event={"ID":"2a80fd3e-aa35-43ea-82a3-54752beddf4b","Type":"ContainerDied","Data":"7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848"}
Jul 24 13:47:25.029275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:25.029212    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5b85e82e325c2c19291d0cc65f831cb08fd41bce35acbb53a7742ffb2d9848"
Jul 24 13:47:36.690363 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:36.690327    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:36.690861 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:36.690532    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" podUID="be2262be-d0a5-4803-81d6-07a06b46e90e" containerName="maas-api" containerID="cri-o://4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e" gracePeriod=30
Jul 24 13:47:36.746394 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:36.746362    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:36.746596 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:36.746570    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" podUID="94bac958-6515-4871-b031-96c461f4537b" containerName="payload-pre-processing" containerID="cri-o://154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1" gracePeriod=30
Jul 24 13:47:36.992652 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:36.992627    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:37.023676 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.023648    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:37.092205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.092169    2556 generic.go:356] "Generic (PLEG): container finished" podID="94bac958-6515-4871-b031-96c461f4537b" containerID="154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1" exitCode=0
Jul 24 13:47:37.092401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.092214    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"
Jul 24 13:47:37.092401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.092240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" event={"ID":"94bac958-6515-4871-b031-96c461f4537b","Type":"ContainerDied","Data":"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"}
Jul 24 13:47:37.092401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.092295    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn" event={"ID":"94bac958-6515-4871-b031-96c461f4537b","Type":"ContainerDied","Data":"a63966fbe8c62a07915d03de03b1dacc418dbb3a18cdccce177af3c55beff915"}
Jul 24 13:47:37.092401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.092324    2556 scope.go:117] "RemoveContainer" containerID="154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"
Jul 24 13:47:37.093499 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.093468    2556 generic.go:356] "Generic (PLEG): container finished" podID="be2262be-d0a5-4803-81d6-07a06b46e90e" containerID="4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e" exitCode=0
Jul 24 13:47:37.093619 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.093513    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"
Jul 24 13:47:37.093619 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.093507    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" event={"ID":"be2262be-d0a5-4803-81d6-07a06b46e90e","Type":"ContainerDied","Data":"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"}
Jul 24 13:47:37.093697 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.093641    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6" event={"ID":"be2262be-d0a5-4803-81d6-07a06b46e90e","Type":"ContainerDied","Data":"efb086ea891fac71941850362e8ac7e573f8b21dad2ee6f65f750e6a71d4664e"}
Jul 24 13:47:37.096081 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.096061    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") pod \"be2262be-d0a5-4803-81d6-07a06b46e90e\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") "
Jul 24 13:47:37.096151 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.096121    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca\") pod \"be2262be-d0a5-4803-81d6-07a06b46e90e\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") "
Jul 24 13:47:37.096237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.096219    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crp56\" (UniqueName: \"kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56\") pod \"be2262be-d0a5-4803-81d6-07a06b46e90e\" (UID: \"be2262be-d0a5-4803-81d6-07a06b46e90e\") "
Jul 24 13:47:37.096482 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.096456    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be2262be-d0a5-4803-81d6-07a06b46e90e" (UID: "be2262be-d0a5-4803-81d6-07a06b46e90e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:47:37.096611 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.096595    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2262be-d0a5-4803-81d6-07a06b46e90e-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:37.098448 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.098422    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be2262be-d0a5-4803-81d6-07a06b46e90e" (UID: "be2262be-d0a5-4803-81d6-07a06b46e90e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:47:37.098573 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.098541    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56" (OuterVolumeSpecName: "kube-api-access-crp56") pod "be2262be-d0a5-4803-81d6-07a06b46e90e" (UID: "be2262be-d0a5-4803-81d6-07a06b46e90e"). InnerVolumeSpecName "kube-api-access-crp56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:47:37.102115 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.102092    2556 scope.go:117] "RemoveContainer" containerID="154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"
Jul 24 13:47:37.102441 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:47:37.102416    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1\": container with ID starting with 154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1 not found: ID does not exist" containerID="154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"
Jul 24 13:47:37.102510 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.102453    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1"} err="failed to get container status \"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1\": rpc error: code = NotFound desc = could not find container \"154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1\": container with ID starting with 154a25fcb45b6b00a1460e63bd2232b9ab481e7b7ca259f69febc47467a4ffb1 not found: ID does not exist"
Jul 24 13:47:37.102510 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.102470    2556 scope.go:117] "RemoveContainer" containerID="4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"
Jul 24 13:47:37.110814 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.110793    2556 scope.go:117] "RemoveContainer" containerID="4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"
Jul 24 13:47:37.111188 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:47:37.111159    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e\": container with ID starting with 4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e not found: ID does not exist" containerID="4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"
Jul 24 13:47:37.111280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.111196    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e"} err="failed to get container status \"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e\": rpc error: code = NotFound desc = could not find container \"4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e\": container with ID starting with 4bb07c7eff3698d9605b80b8cf4e98a1c90728c6c2d3d9fc997965534e3bec1e not found: ID does not exist"
Jul 24 13:47:37.197581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.197543    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume\") pod \"94bac958-6515-4871-b031-96c461f4537b\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") "
Jul 24 13:47:37.197581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.197587    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbh9x\" (UniqueName: \"kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x\") pod \"94bac958-6515-4871-b031-96c461f4537b\" (UID: \"94bac958-6515-4871-b031-96c461f4537b\") "
Jul 24 13:47:37.197806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.197779    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be2262be-d0a5-4803-81d6-07a06b46e90e-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:37.197806 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.197797    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crp56\" (UniqueName: \"kubernetes.io/projected/be2262be-d0a5-4803-81d6-07a06b46e90e-kube-api-access-crp56\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:37.197965 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.197910    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94bac958-6515-4871-b031-96c461f4537b" (UID: "94bac958-6515-4871-b031-96c461f4537b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:47:37.199825 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.199803    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x" (OuterVolumeSpecName: "kube-api-access-dbh9x") pod "94bac958-6515-4871-b031-96c461f4537b" (UID: "94bac958-6515-4871-b031-96c461f4537b"). InnerVolumeSpecName "kube-api-access-dbh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:47:37.298887 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.298856    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94bac958-6515-4871-b031-96c461f4537b-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:37.298887 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.298882    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbh9x\" (UniqueName: \"kubernetes.io/projected/94bac958-6515-4871-b031-96c461f4537b-kube-api-access-dbh9x\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:37.407706 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.407666    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:37.413791 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.413763    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-502742-6d868cc856-dgmrn"]
Jul 24 13:47:37.423782 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.423751    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:37.428191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:37.428165    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-502742-9d6b6696-2jpz6"]
Jul 24 13:47:39.341261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:39.341228    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94bac958-6515-4871-b031-96c461f4537b" path="/var/lib/kubelet/pods/94bac958-6515-4871-b031-96c461f4537b/volumes"
Jul 24 13:47:39.341632 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:39.341597    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2262be-d0a5-4803-81d6-07a06b46e90e" path="/var/lib/kubelet/pods/be2262be-d0a5-4803-81d6-07a06b46e90e/volumes"
Jul 24 13:47:40.939893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:40.939858    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"]
Jul 24 13:47:40.942709 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:40.942676    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-502742-2jt44"]
Jul 24 13:47:41.341916 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:41.341881    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a80fd3e-aa35-43ea-82a3-54752beddf4b" path="/var/lib/kubelet/pods/2a80fd3e-aa35-43ea-82a3-54752beddf4b/volumes"
Jul 24 13:47:47.812280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812241    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"]
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812585    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bac958-6515-4871-b031-96c461f4537b" containerName="payload-pre-processing"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812596    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bac958-6515-4871-b031-96c461f4537b" containerName="payload-pre-processing"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812614    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2262be-d0a5-4803-81d6-07a06b46e90e" containerName="maas-api"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812619    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2262be-d0a5-4803-81d6-07a06b46e90e" containerName="maas-api"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812626    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a80fd3e-aa35-43ea-82a3-54752beddf4b" containerName="revoke-keys"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812631    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a80fd3e-aa35-43ea-82a3-54752beddf4b" containerName="revoke-keys"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812708    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bac958-6515-4871-b031-96c461f4537b" containerName="payload-pre-processing"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812716    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2262be-d0a5-4803-81d6-07a06b46e90e" containerName="maas-api"
Jul 24 13:47:47.812764 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.812724    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a80fd3e-aa35-43ea-82a3-54752beddf4b" containerName="revoke-keys"
Jul 24 13:47:47.815450 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.815432    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:47.817702 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.817683    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:47:47.822674 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.822250    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"]
Jul 24 13:47:47.896867 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.896824    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-799cec-w2kvr\" (UID: \"866d8c1a-3314-42fb-b6ee-a655ad446e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:47.998079 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.998036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-799cec-w2kvr\" (UID: \"866d8c1a-3314-42fb-b6ee-a655ad446e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:47.998623 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:47.998604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-799cec-w2kvr\" (UID: \"866d8c1a-3314-42fb-b6ee-a655ad446e03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:48.126267 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:48.126174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:48.267077 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:48.267045    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"]
Jul 24 13:47:48.269659 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:47:48.269628    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866d8c1a_3314_42fb_b6ee_a655ad446e03.slice/crio-17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d WatchSource:0}: Error finding container 17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d: Status 404 returned error can't find the container with id 17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d
Jul 24 13:47:49.157240 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:49.157150    2556 generic.go:356] "Generic (PLEG): container finished" podID="866d8c1a-3314-42fb-b6ee-a655ad446e03" containerID="2be1fcd7cf3e337cb294be470f5175f532968633eb25ced9c9d191e1bcd4cbec" exitCode=0
Jul 24 13:47:49.157600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:49.157243    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr" event={"ID":"866d8c1a-3314-42fb-b6ee-a655ad446e03","Type":"ContainerDied","Data":"2be1fcd7cf3e337cb294be470f5175f532968633eb25ced9c9d191e1bcd4cbec"}
Jul 24 13:47:49.157600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:49.157308    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr" event={"ID":"866d8c1a-3314-42fb-b6ee-a655ad446e03","Type":"ContainerStarted","Data":"17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d"}
Jul 24 13:47:50.295733 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:50.295710    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:50.319237 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:50.319205    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca\") pod \"866d8c1a-3314-42fb-b6ee-a655ad446e03\" (UID: \"866d8c1a-3314-42fb-b6ee-a655ad446e03\") "
Jul 24 13:47:50.319545 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:50.319524    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "866d8c1a-3314-42fb-b6ee-a655ad446e03" (UID: "866d8c1a-3314-42fb-b6ee-a655ad446e03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:47:50.420783 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:50.420691    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/866d8c1a-3314-42fb-b6ee-a655ad446e03-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:47:51.168695 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:51.168662    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"
Jul 24 13:47:51.168883 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:51.168667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr" event={"ID":"866d8c1a-3314-42fb-b6ee-a655ad446e03","Type":"ContainerDied","Data":"17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d"}
Jul 24 13:47:51.168883 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:47:51.168783    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17188b433134a5f227ff79085955a53e3f716a2e5c24b8bb620a1dca8580b69d"
Jul 24 13:48:03.233433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.233396    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:48:03.234068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.233646    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" podUID="8e511262-3727-489a-bb1c-d829661b3716" containerName="maas-api" containerID="cri-o://89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee" gracePeriod=30
Jul 24 13:48:03.479579 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.479553    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:48:03.537515 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.537485    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8g6w\" (UniqueName: \"kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w\") pod \"8e511262-3727-489a-bb1c-d829661b3716\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") "
Jul 24 13:48:03.537694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.537550    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca\") pod \"8e511262-3727-489a-bb1c-d829661b3716\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") "
Jul 24 13:48:03.537694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.537602    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls\") pod \"8e511262-3727-489a-bb1c-d829661b3716\" (UID: \"8e511262-3727-489a-bb1c-d829661b3716\") "
Jul 24 13:48:03.537986 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.537954    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e511262-3727-489a-bb1c-d829661b3716" (UID: "8e511262-3727-489a-bb1c-d829661b3716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:48:03.539950 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.539899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e511262-3727-489a-bb1c-d829661b3716" (UID: "8e511262-3727-489a-bb1c-d829661b3716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:48:03.540060 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.539987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w" (OuterVolumeSpecName: "kube-api-access-n8g6w") pod "8e511262-3727-489a-bb1c-d829661b3716" (UID: "8e511262-3727-489a-bb1c-d829661b3716"). InnerVolumeSpecName "kube-api-access-n8g6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:48:03.638871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.638835    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e511262-3727-489a-bb1c-d829661b3716-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:48:03.638871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.638864    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e511262-3727-489a-bb1c-d829661b3716-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:48:03.638871 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:03.638874    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8g6w\" (UniqueName: \"kubernetes.io/projected/8e511262-3727-489a-bb1c-d829661b3716-kube-api-access-n8g6w\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:48:04.238997 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.238962    2556 generic.go:356] "Generic (PLEG): container finished" podID="8e511262-3727-489a-bb1c-d829661b3716" containerID="89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee" exitCode=0
Jul 24 13:48:04.239409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.239066    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"
Jul 24 13:48:04.239409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.239060    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" event={"ID":"8e511262-3727-489a-bb1c-d829661b3716","Type":"ContainerDied","Data":"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"}
Jul 24 13:48:04.239409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.239191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95" event={"ID":"8e511262-3727-489a-bb1c-d829661b3716","Type":"ContainerDied","Data":"8130b74b61405d7746609a7cd0d668da6d0ed4dc4565830f243cab9b175596d3"}
Jul 24 13:48:04.239409 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.239222    2556 scope.go:117] "RemoveContainer" containerID="89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"
Jul 24 13:48:04.249468 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.249450    2556 scope.go:117] "RemoveContainer" containerID="89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"
Jul 24 13:48:04.249759 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:48:04.249740    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee\": container with ID starting with 89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee not found: ID does not exist" containerID="89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"
Jul 24 13:48:04.249809 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.249771    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee"} err="failed to get container status \"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee\": rpc error: code = NotFound desc = could not find container \"89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee\": container with ID starting with 89837c58e770376ae15a07b90a9967ab435c7a0262f3949159e10978d21d43ee not found: ID does not exist"
Jul 24 13:48:04.260395 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.260369    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:48:04.264176 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:04.264154    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-799cec-5c4998c94c-b9n95"]
Jul 24 13:48:05.341550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:05.341505    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e511262-3727-489a-bb1c-d829661b3716" path="/var/lib/kubelet/pods/8e511262-3727-489a-bb1c-d829661b3716/volumes"
Jul 24 13:48:07.903534 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:07.903499    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"]
Jul 24 13:48:07.906095 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:07.906073    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-799cec-w2kvr"]
Jul 24 13:48:09.340469 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:09.340435    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="866d8c1a-3314-42fb-b6ee-a655ad446e03" path="/var/lib/kubelet/pods/866d8c1a-3314-42fb-b6ee-a655ad446e03/volumes"
Jul 24 13:48:18.653754 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.653718    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654126    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="866d8c1a-3314-42fb-b6ee-a655ad446e03" containerName="revoke-keys"
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654140    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="866d8c1a-3314-42fb-b6ee-a655ad446e03" containerName="revoke-keys"
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654150    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e511262-3727-489a-bb1c-d829661b3716" containerName="maas-api"
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654155    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e511262-3727-489a-bb1c-d829661b3716" containerName="maas-api"
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654206    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="866d8c1a-3314-42fb-b6ee-a655ad446e03" containerName="revoke-keys"
Jul 24 13:48:18.654277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.654216    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e511262-3727-489a-bb1c-d829661b3716" containerName="maas-api"
Jul 24 13:48:18.657191 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.657174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.659818 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.659797    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-934481-serving-cert\""
Jul 24 13:48:18.659961 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.659894    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9lpb\""
Jul 24 13:48:18.668264 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.668241    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:48:18.776848 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.776811    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.777054 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.776877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mt5b\" (UniqueName: \"kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.777054 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.776958    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.877951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.877894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mt5b\" (UniqueName: \"kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.877951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.877950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.878183 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.878011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.878755 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.878725    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.880644 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.880627    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.886304 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.886283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mt5b\" (UniqueName: \"kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b\") pod \"maas-api-e2e-shared-a-934481-699d76cd54-k4bsz\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:18.968183 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:18.968095    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:19.105115 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:19.105088    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:48:19.107811 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:48:19.107789    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e9208d_5d19_4add_bebc_deec0dd8e161.slice/crio-52b632b72fe78429386e1f0c28f0b4d4b46c72c978b46a4a439edade65351b78 WatchSource:0}: Error finding container 52b632b72fe78429386e1f0c28f0b4d4b46c72c978b46a4a439edade65351b78: Status 404 returned error can't find the container with id 52b632b72fe78429386e1f0c28f0b4d4b46c72c978b46a4a439edade65351b78
Jul 24 13:48:19.320098 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:19.320059    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" event={"ID":"81e9208d-5d19-4add-bebc-deec0dd8e161","Type":"ContainerStarted","Data":"52b632b72fe78429386e1f0c28f0b4d4b46c72c978b46a4a439edade65351b78"}
Jul 24 13:48:20.334894 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:20.334848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" event={"ID":"81e9208d-5d19-4add-bebc-deec0dd8e161","Type":"ContainerStarted","Data":"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"}
Jul 24 13:48:20.335410 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:20.335250    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:20.353391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:20.353341    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" podStartSLOduration=1.962858325 podStartE2EDuration="2.353325919s" totalImagesPullingTime="390.467594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:20.351862605 +0000 UTC m=+2319.576345933" watchObservedRunningTime="2026-07-24 13:48:20.353325919 +0000 UTC m=+2319.577809245"
Jul 24 13:48:26.345474 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.345448    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:48:26.657207 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.657121    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:48:26.666291 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.665563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.669205 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.669182    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0a0df7-serving-cert\""
Jul 24 13:48:26.674754 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.674725    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:48:26.851416 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.851376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5pw\" (UniqueName: \"kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.851636 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.851458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.851636 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.851485    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.952726 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.952634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.952726 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.952673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.952920 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.952741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5pw\" (UniqueName: \"kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.952920 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:48:26.952786    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-serving-cert: secret "maas-api-e2e-shared-b-0a0df7-serving-cert" not found
Jul 24 13:48:26.952920 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:48:26.952873    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls podName:9ab4676e-738e-4812-bc07-3a5a14eab39e nodeName:}" failed. No retries permitted until 2026-07-24 13:48:27.452856018 +0000 UTC m=+2326.677339322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls") pod "maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" (UID: "9ab4676e-738e-4812-bc07-3a5a14eab39e") : secret "maas-api-e2e-shared-b-0a0df7-serving-cert" not found
Jul 24 13:48:26.953448 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.953425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:26.961965 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:26.961919    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5pw\" (UniqueName: \"kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:27.457581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.457543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:27.460129 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.460100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") pod \"maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:27.489076 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.489038    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:48:27.492472 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.492457    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.494844 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.494817    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-0a0df7\""
Jul 24 13:48:27.494983 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.494826    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-0a0df7-dockercfg-scmx2\""
Jul 24 13:48:27.504810 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.504787    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:48:27.558369 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.558328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq56l\" (UniqueName: \"kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.558536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.558508    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.580188 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.580157    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:27.659324 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.659294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq56l\" (UniqueName: \"kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.659463 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.659443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.660166 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.660147    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.668836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.668810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq56l\" (UniqueName: \"kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l\") pod \"payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.741523 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.741496    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:48:27.743908 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:48:27.743876    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab4676e_738e_4812_bc07_3a5a14eab39e.slice/crio-f9280a51241b5439d0d690fd59f93c9a8746638fba992c0e51aae5fd0ce37702 WatchSource:0}: Error finding container f9280a51241b5439d0d690fd59f93c9a8746638fba992c0e51aae5fd0ce37702: Status 404 returned error can't find the container with id f9280a51241b5439d0d690fd59f93c9a8746638fba992c0e51aae5fd0ce37702
Jul 24 13:48:27.803624 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.803584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:27.940857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:27.940828    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:48:27.943102 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:48:27.943073    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600156c9_59d2_4948_8bf2_5b0b045732cc.slice/crio-dcd8aa1b57fb047485c2d750212a968e3fd7e487244638c19677c59b1e9c85b6 WatchSource:0}: Error finding container dcd8aa1b57fb047485c2d750212a968e3fd7e487244638c19677c59b1e9c85b6: Status 404 returned error can't find the container with id dcd8aa1b57fb047485c2d750212a968e3fd7e487244638c19677c59b1e9c85b6
Jul 24 13:48:28.375875 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.375827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" event={"ID":"9ab4676e-738e-4812-bc07-3a5a14eab39e","Type":"ContainerStarted","Data":"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"}
Jul 24 13:48:28.376102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.375889    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:28.376102 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.375910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" event={"ID":"9ab4676e-738e-4812-bc07-3a5a14eab39e","Type":"ContainerStarted","Data":"f9280a51241b5439d0d690fd59f93c9a8746638fba992c0e51aae5fd0ce37702"}
Jul 24 13:48:28.377275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.377248    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" event={"ID":"600156c9-59d2-4948-8bf2-5b0b045732cc","Type":"ContainerStarted","Data":"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"}
Jul 24 13:48:28.377386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.377299    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" event={"ID":"600156c9-59d2-4948-8bf2-5b0b045732cc","Type":"ContainerStarted","Data":"dcd8aa1b57fb047485c2d750212a968e3fd7e487244638c19677c59b1e9c85b6"}
Jul 24 13:48:28.377386 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.377341    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:28.395635 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.395579    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" podStartSLOduration=2.031832067 podStartE2EDuration="2.395563753s" totalImagesPullingTime="363.731686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:28.393385152 +0000 UTC m=+2327.617868489" watchObservedRunningTime="2026-07-24 13:48:28.395563753 +0000 UTC m=+2327.620047085"
Jul 24 13:48:28.410487 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:28.410439    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" podStartSLOduration=1.410421589 podStartE2EDuration="1.410421589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:28.409817791 +0000 UTC m=+2327.634301111" watchObservedRunningTime="2026-07-24 13:48:28.410421589 +0000 UTC m=+2327.634904928"
Jul 24 13:48:34.387199 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:34.387123    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:48:39.383832 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:39.383799    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:48:42.015374 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:42.015338    2556 scope.go:117] "RemoveContainer" containerID="905ec63cab4c6523e7a2c5a8adf644ca00f0d3ae3647a60bf5e1b761095e0c38"
Jul 24 13:48:50.914180 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.914136    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:48:50.917837 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.917816    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.921379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.921354    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"default-dockercfg-489nj\""
Jul 24 13:48:50.921605 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.921396    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"auth-test-model-934481-kserve-self-signed-certs\""
Jul 24 13:48:50.921842 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.921816    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"kube-root-ca.crt\""
Jul 24 13:48:50.922026 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.921421    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"openshift-service-ca.crt\""
Jul 24 13:48:50.928695 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.928672    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:48:50.959033 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.959179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.959179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.959179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqp8\" (UniqueName: \"kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.959179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:50.959321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:50.959224    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqp8\" (UniqueName: \"kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060625 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060832 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060808    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060876 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.060921 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.060879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.062893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.062862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.063216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.063195    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.072037 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.072017    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqp8\" (UniqueName: \"kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8\") pod \"auth-test-model-934481-kserve-6f75b7c68c-nhs4r\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") " pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.229857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.229760    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:48:51.367069 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.367038    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:48:51.369760 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:48:51.369735    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c4baa8d_2259_477b_a75d_e51f8988896a.slice/crio-dea095ff416b89ffbe5eba5625fc1427be598bc593131336b464ebc16090529b WatchSource:0}: Error finding container dea095ff416b89ffbe5eba5625fc1427be598bc593131336b464ebc16090529b: Status 404 returned error can't find the container with id dea095ff416b89ffbe5eba5625fc1427be598bc593131336b464ebc16090529b
Jul 24 13:48:51.498692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.498609    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" event={"ID":"9c4baa8d-2259-477b-a75d-e51f8988896a","Type":"ContainerStarted","Data":"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"}
Jul 24 13:48:51.498692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.498659    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" event={"ID":"9c4baa8d-2259-477b-a75d-e51f8988896a","Type":"ContainerStarted","Data":"dea095ff416b89ffbe5eba5625fc1427be598bc593131336b464ebc16090529b"}
Jul 24 13:48:51.517470 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:48:51.517417    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" podStartSLOduration=1.517400788 podStartE2EDuration="1.517400788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:51.51600925 +0000 UTC m=+2350.740492578" watchObservedRunningTime="2026-07-24 13:48:51.517400788 +0000 UTC m=+2350.741884159"
Jul 24 13:49:01.230477 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:01.230437    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:49:01.231006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:01.230497    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:49:01.242371 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:01.242344    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:49:01.560904 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:01.560877    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:49:41.346234 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:41.346208    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:49:41.350631 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:49:41.350609    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:51:17.221006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:17.220971    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:51:30.618480 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.618442    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:51:30.621727 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.621711    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.623980 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.623961    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"sub-test-model-934481-kserve-self-signed-certs\""
Jul 24 13:51:30.632675 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.632651    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:51:30.734473 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.734634 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734477    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.734634 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.734634 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjk8j\" (UniqueName: \"kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.734738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734640    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.734738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.734659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.835827 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.835792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.835827 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.835832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjk8j\" (UniqueName: \"kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.835863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.835893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.835975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.836000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836274 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.836253    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836337 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.836312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.836574 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.836540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.838480 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.838461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.838562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.838542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.844688 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.844665    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjk8j\" (UniqueName: \"kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j\") pod \"sub-test-model-934481-kserve-5f6c6484cc-7khql\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") " pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:30.931018 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:30.930947    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:31.064216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:31.064185    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:51:31.067641 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:51:31.067617    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5336e37_54fe_44f0_9939_50f91754a035.slice/crio-ccce0556c4ce80ff9b8bb2af247c1fa614d937759c358c54b444dabfc49de230 WatchSource:0}: Error finding container ccce0556c4ce80ff9b8bb2af247c1fa614d937759c358c54b444dabfc49de230: Status 404 returned error can't find the container with id ccce0556c4ce80ff9b8bb2af247c1fa614d937759c358c54b444dabfc49de230
Jul 24 13:51:31.329134 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:31.329094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" event={"ID":"f5336e37-54fe-44f0-9939-50f91754a035","Type":"ContainerStarted","Data":"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"}
Jul 24 13:51:31.329316 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:31.329143    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" event={"ID":"f5336e37-54fe-44f0-9939-50f91754a035","Type":"ContainerStarted","Data":"ccce0556c4ce80ff9b8bb2af247c1fa614d937759c358c54b444dabfc49de230"}
Jul 24 13:51:31.352477 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:31.352393    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" podStartSLOduration=1.3523741519999999 podStartE2EDuration="1.352374152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:51:31.351895312 +0000 UTC m=+2510.576378646" watchObservedRunningTime="2026-07-24 13:51:31.352374152 +0000 UTC m=+2510.576857481"
Jul 24 13:51:32.240365 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.240302    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" podUID="9c4baa8d-2259-477b-a75d-e51f8988896a" containerName="main" containerID="cri-o://aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f" gracePeriod=45
Jul 24 13:51:32.483025 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.482999    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:51:32.554678 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554653    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.554847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554718    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkqp8\" (UniqueName: \"kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.554847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.554847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554774    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.554847 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554818    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.555106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554848    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs\") pod \"9c4baa8d-2259-477b-a75d-e51f8988896a\" (UID: \"9c4baa8d-2259-477b-a75d-e51f8988896a\") "
Jul 24 13:51:32.555106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.554907    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:51:32.555106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.555049    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home" (OuterVolumeSpecName: "home") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:51:32.555106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.555096    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:51:32.555278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.555229    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:32.555278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.555252    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:32.555278 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.555267    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:32.557189 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.557164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:51:32.557274 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.557212    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8" (OuterVolumeSpecName: "kube-api-access-vkqp8") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "kube-api-access-vkqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:51:32.557274 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.557247    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm" (OuterVolumeSpecName: "dshm") pod "9c4baa8d-2259-477b-a75d-e51f8988896a" (UID: "9c4baa8d-2259-477b-a75d-e51f8988896a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:51:32.656003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.655974    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqp8\" (UniqueName: \"kubernetes.io/projected/9c4baa8d-2259-477b-a75d-e51f8988896a-kube-api-access-vkqp8\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:32.656003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.655999    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4baa8d-2259-477b-a75d-e51f8988896a-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:32.656003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:32.656009    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4baa8d-2259-477b-a75d-e51f8988896a-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:51:33.341843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.341806    2556 generic.go:356] "Generic (PLEG): container finished" podID="9c4baa8d-2259-477b-a75d-e51f8988896a" containerID="aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f" exitCode=0
Jul 24 13:51:33.342241 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.341917    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"
Jul 24 13:51:33.342241 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.342081    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" event={"ID":"9c4baa8d-2259-477b-a75d-e51f8988896a","Type":"ContainerDied","Data":"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"}
Jul 24 13:51:33.342241 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.342115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r" event={"ID":"9c4baa8d-2259-477b-a75d-e51f8988896a","Type":"ContainerDied","Data":"dea095ff416b89ffbe5eba5625fc1427be598bc593131336b464ebc16090529b"}
Jul 24 13:51:33.342241 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.342141    2556 scope.go:117] "RemoveContainer" containerID="aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"
Jul 24 13:51:33.355465 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.355448    2556 scope.go:117] "RemoveContainer" containerID="aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"
Jul 24 13:51:33.355740 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:51:33.355720    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f\": container with ID starting with aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f not found: ID does not exist" containerID="aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"
Jul 24 13:51:33.355791 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.355750    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f"} err="failed to get container status \"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f\": rpc error: code = NotFound desc = could not find container \"aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f\": container with ID starting with aca3358a772982f62df5c25c4b4d71717c4c37fee7abd0d2c7b11e0af25efc0f not found: ID does not exist"
Jul 24 13:51:33.366605 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.366583    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:51:33.372408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:33.372389    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-934481/auth-test-model-934481-kserve-6f75b7c68c-nhs4r"]
Jul 24 13:51:35.340661 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:35.340625    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4baa8d-2259-477b-a75d-e51f8988896a" path="/var/lib/kubelet/pods/9c4baa8d-2259-477b-a75d-e51f8988896a/volumes"
Jul 24 13:51:40.931123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:40.931082    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:40.931575 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:40.931140    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:40.944371 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:40.944346    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:41.392415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:41.392387    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:51:42.052435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:42.052404    2556 scope.go:117] "RemoveContainer" containerID="1fec58bb14c16d43c74cedac02edc63c7a2d347d83ca738c7c0100c5d68972c4"
Jul 24 13:51:42.060773 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:42.060755    2556 scope.go:117] "RemoveContainer" containerID="dc8ef553100e37b187ffa92f95960a0ffb69b99bcfd7709be6ccd2fb7fac9259"
Jul 24 13:51:47.128165 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.128129    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:51:47.128626 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.128540    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4baa8d-2259-477b-a75d-e51f8988896a" containerName="main"
Jul 24 13:51:47.128626 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.128554    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4baa8d-2259-477b-a75d-e51f8988896a" containerName="main"
Jul 24 13:51:47.128699 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.128637    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4baa8d-2259-477b-a75d-e51f8988896a" containerName="main"
Jul 24 13:51:47.131609 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.131587    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.134683 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.134655    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"sub-test-model-0a0df7-kserve-self-signed-certs\""
Jul 24 13:51:47.134796 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.134678    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"kube-root-ca.crt\""
Jul 24 13:51:47.134796 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.134655    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"default-dockercfg-md425\""
Jul 24 13:51:47.134796 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.134689    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"openshift-service-ca.crt\""
Jul 24 13:51:47.140451 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.140426    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:51:47.295058 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.295458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwp8\" (UniqueName: \"kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.295458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.295458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.295458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.295458 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.295303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396087 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.395997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396087 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwp8\" (UniqueName: \"kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396305 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396476 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396476 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396407    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396706 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396674    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.396805 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.396695    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.398560 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.398535    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.399022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.399002    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.404988 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.404960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwp8\" (UniqueName: \"kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8\") pod \"sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") " pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.442202 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.442171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:47.580074 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:47.580011    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:51:47.583270 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:51:47.583226    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa40664_8a5e_428b_8828_e42509c01742.slice/crio-3eb0f7629b9340c4a11fe94a96ecedd9e406a8704e08ebad41640a60ec46ff00 WatchSource:0}: Error finding container 3eb0f7629b9340c4a11fe94a96ecedd9e406a8704e08ebad41640a60ec46ff00: Status 404 returned error can't find the container with id 3eb0f7629b9340c4a11fe94a96ecedd9e406a8704e08ebad41640a60ec46ff00
Jul 24 13:51:48.418433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:48.418381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" event={"ID":"5fa40664-8a5e-428b-8828-e42509c01742","Type":"ContainerStarted","Data":"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"}
Jul 24 13:51:48.418433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:48.418432    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" event={"ID":"5fa40664-8a5e-428b-8828-e42509c01742","Type":"ContainerStarted","Data":"3eb0f7629b9340c4a11fe94a96ecedd9e406a8704e08ebad41640a60ec46ff00"}
Jul 24 13:51:48.435738 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:48.435695    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" podStartSLOduration=1.4356815680000001 podStartE2EDuration="1.435681568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:51:48.434859253 +0000 UTC m=+2527.659342580" watchObservedRunningTime="2026-07-24 13:51:48.435681568 +0000 UTC m=+2527.660164894"
Jul 24 13:51:57.442281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:57.442244    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:57.442281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:57.442288    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:57.454954 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:57.454911    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:51:57.472688 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:51:57.472657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:52:28.336655 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:28.336619    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:52:29.297703 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:29.297652    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:52:42.085741 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:42.085704    2556 scope.go:117] "RemoveContainer" containerID="0cacbc9633fe59f895ef547e1aada149a5905fbe25fbbb60a5c433d9e2ca2b62"
Jul 24 13:52:43.355663 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.355622    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" podUID="f5336e37-54fe-44f0-9939-50f91754a035" containerName="main" containerID="cri-o://858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26" gracePeriod=45
Jul 24 13:52:43.605253 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.605228    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:52:43.699716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.699630    2556 generic.go:356] "Generic (PLEG): container finished" podID="f5336e37-54fe-44f0-9939-50f91754a035" containerID="858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26" exitCode=0
Jul 24 13:52:43.699716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.699667    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"
Jul 24 13:52:43.699716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.699680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" event={"ID":"f5336e37-54fe-44f0-9939-50f91754a035","Type":"ContainerDied","Data":"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"}
Jul 24 13:52:43.699716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.699718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql" event={"ID":"f5336e37-54fe-44f0-9939-50f91754a035","Type":"ContainerDied","Data":"ccce0556c4ce80ff9b8bb2af247c1fa614d937759c358c54b444dabfc49de230"}
Jul 24 13:52:43.700016 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.699735    2556 scope.go:117] "RemoveContainer" containerID="858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"
Jul 24 13:52:43.708455 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.708430    2556 scope.go:117] "RemoveContainer" containerID="858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"
Jul 24 13:52:43.708716 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:52:43.708696    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26\": container with ID starting with 858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26 not found: ID does not exist" containerID="858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"
Jul 24 13:52:43.708766 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.708727    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26"} err="failed to get container status \"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26\": rpc error: code = NotFound desc = could not find container \"858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26\": container with ID starting with 858e3d6972b175362ababc1e983bc2844a1f0de966beb6c70c229c2111654a26 not found: ID does not exist"
Jul 24 13:52:43.709991 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.709973    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710060 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710001    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710078    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjk8j\" (UniqueName: \"kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710183 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710139    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710194    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710212    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home\") pod \"f5336e37-54fe-44f0-9939-50f91754a035\" (UID: \"f5336e37-54fe-44f0-9939-50f91754a035\") "
Jul 24 13:52:43.710330 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710251    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:43.710489 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710459    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:43.710489 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710473    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:43.710656 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.710551    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home" (OuterVolumeSpecName: "home") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:43.712259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.712234    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm" (OuterVolumeSpecName: "dshm") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:43.712345 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.712285    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j" (OuterVolumeSpecName: "kube-api-access-gjk8j") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "kube-api-access-gjk8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:52:43.712345 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.712330    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5336e37-54fe-44f0-9939-50f91754a035" (UID: "f5336e37-54fe-44f0-9939-50f91754a035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:52:43.811999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.811967    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5336e37-54fe-44f0-9939-50f91754a035-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:43.811999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.811997    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjk8j\" (UniqueName: \"kubernetes.io/projected/f5336e37-54fe-44f0-9939-50f91754a035-kube-api-access-gjk8j\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:43.811999 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.812007    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:43.812216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.812015    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:43.812216 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:43.812022    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5336e37-54fe-44f0-9939-50f91754a035-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.022021 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.021988    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:52:44.024801 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.024777    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-934481/sub-test-model-934481-kserve-5f6c6484cc-7khql"]
Jul 24 13:52:44.317390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.317352    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" podUID="5fa40664-8a5e-428b-8828-e42509c01742" containerName="main" containerID="cri-o://32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058" gracePeriod=45
Jul 24 13:52:44.570083 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.570023    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:52:44.620100 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620100 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620105    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620158    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620204    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mwp8\" (UniqueName: \"kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620221    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620256    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home\") pod \"5fa40664-8a5e-428b-8828-e42509c01742\" (UID: \"5fa40664-8a5e-428b-8828-e42509c01742\") "
Jul 24 13:52:44.620532 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620370    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:44.620592 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620575    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home" (OuterVolumeSpecName: "home") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:44.620652 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620584    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:44.620652 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.620628    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.622504 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.622466    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm" (OuterVolumeSpecName: "dshm") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:52:44.622504 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.622479    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8" (OuterVolumeSpecName: "kube-api-access-2mwp8") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "kube-api-access-2mwp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:52:44.622656 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.622553    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fa40664-8a5e-428b-8828-e42509c01742" (UID: "5fa40664-8a5e-428b-8828-e42509c01742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:52:44.705236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.705205    2556 generic.go:356] "Generic (PLEG): container finished" podID="5fa40664-8a5e-428b-8828-e42509c01742" containerID="32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058" exitCode=0
Jul 24 13:52:44.705417 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.705251    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"
Jul 24 13:52:44.705417 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.705296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" event={"ID":"5fa40664-8a5e-428b-8828-e42509c01742","Type":"ContainerDied","Data":"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"}
Jul 24 13:52:44.705417 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.705339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb" event={"ID":"5fa40664-8a5e-428b-8828-e42509c01742","Type":"ContainerDied","Data":"3eb0f7629b9340c4a11fe94a96ecedd9e406a8704e08ebad41640a60ec46ff00"}
Jul 24 13:52:44.705417 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.705365    2556 scope.go:117] "RemoveContainer" containerID="32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"
Jul 24 13:52:44.714692 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.714676    2556 scope.go:117] "RemoveContainer" containerID="32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"
Jul 24 13:52:44.714987 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:52:44.714963    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058\": container with ID starting with 32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058 not found: ID does not exist" containerID="32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"
Jul 24 13:52:44.715044 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.715002    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058"} err="failed to get container status \"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058\": rpc error: code = NotFound desc = could not find container \"32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058\": container with ID starting with 32028ccf78adb61c62594effed9e8152be889e30d52a908b5841c2fb10877058 not found: ID does not exist"
Jul 24 13:52:44.721714 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.721696    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa40664-8a5e-428b-8828-e42509c01742-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.721802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.721722    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwp8\" (UniqueName: \"kubernetes.io/projected/5fa40664-8a5e-428b-8828-e42509c01742-kube-api-access-2mwp8\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.721802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.721736    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.721802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.721753    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.721802 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.721765    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa40664-8a5e-428b-8828-e42509c01742-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:52:44.728016 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.727996    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:52:44.731593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:44.731571    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/sub-test-model-0a0df7-kserve-5f4fbd889f-hhgnb"]
Jul 24 13:52:45.340758 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:45.340722    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa40664-8a5e-428b-8828-e42509c01742" path="/var/lib/kubelet/pods/5fa40664-8a5e-428b-8828-e42509c01742/volumes"
Jul 24 13:52:45.341150 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:45.341136    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5336e37-54fe-44f0-9939-50f91754a035" path="/var/lib/kubelet/pods/f5336e37-54fe-44f0-9939-50f91754a035/volumes"
Jul 24 13:52:50.017778 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.017744    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:52:50.018317 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018304    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa40664-8a5e-428b-8828-e42509c01742" containerName="main"
Jul 24 13:52:50.018390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018323    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa40664-8a5e-428b-8828-e42509c01742" containerName="main"
Jul 24 13:52:50.018390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018340    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5336e37-54fe-44f0-9939-50f91754a035" containerName="main"
Jul 24 13:52:50.018390 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018349    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5336e37-54fe-44f0-9939-50f91754a035" containerName="main"
Jul 24 13:52:50.018541 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018471    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5336e37-54fe-44f0-9939-50f91754a035" containerName="main"
Jul 24 13:52:50.018541 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.018488    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa40664-8a5e-428b-8828-e42509c01742" containerName="main"
Jul 24 13:52:50.021488 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.021469    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.024392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.024369    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"default-dockercfg-489nj\""
Jul 24 13:52:50.024524 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.024369    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"openshift-service-ca.crt\""
Jul 24 13:52:50.024524 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.024446    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"rate-test-model-934481-kserve-self-signed-certs\""
Jul 24 13:52:50.024524 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.024469    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-934481\"/\"kube-root-ca.crt\""
Jul 24 13:52:50.030132 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.030110    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:52:50.071219 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.071359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.071408 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071353    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.071471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr788\" (UniqueName: \"kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.071514 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.071561 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.071520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172234 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr788\" (UniqueName: \"kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172434 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172434 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172359    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172434 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172618 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172618 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172500    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172715 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172715 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172705    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.172782 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.172739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.174959 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.174918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.175131 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.175112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.181585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.181563    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr788\" (UniqueName: \"kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788\") pod \"rate-test-model-934481-kserve-945ddd879-kqtcc\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") " pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.331530 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.331495    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:52:50.464891 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.464816    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:52:50.467048 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:52:50.467019    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeeeeb84_4a6c_4020_9c7e_4bbe7a67781d.slice/crio-0c3745c15898f29b3da00fe161eeb45c8a4f201581a8dfb65425d29fc7cc3368 WatchSource:0}: Error finding container 0c3745c15898f29b3da00fe161eeb45c8a4f201581a8dfb65425d29fc7cc3368: Status 404 returned error can't find the container with id 0c3745c15898f29b3da00fe161eeb45c8a4f201581a8dfb65425d29fc7cc3368
Jul 24 13:52:50.737147 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.737061    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" event={"ID":"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d","Type":"ContainerStarted","Data":"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"}
Jul 24 13:52:50.737147 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.737101    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" event={"ID":"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d","Type":"ContainerStarted","Data":"0c3745c15898f29b3da00fe161eeb45c8a4f201581a8dfb65425d29fc7cc3368"}
Jul 24 13:52:50.754574 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:52:50.754530    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" podStartSLOduration=0.75451411 podStartE2EDuration="754.51411ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:52:50.753077111 +0000 UTC m=+2589.977560451" watchObservedRunningTime="2026-07-24 13:52:50.75451411 +0000 UTC m=+2589.978997437"
Jul 24 13:53:00.332079 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:00.332032    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:53:00.332506 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:00.332317    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:53:00.344426 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:00.344399    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:53:00.797226 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:00.797196    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:53:42.105836 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:42.105806    2556 scope.go:117] "RemoveContainer" containerID="3b0cec819e77c26412c555084fb09ef3c6b7142e60238f917a9e65ef5c79b2b8"
Jul 24 13:53:50.024528 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.024494    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:53:50.028373 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.028353    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.030762 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.030740    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"openshift-service-ca.crt\""
Jul 24 13:53:50.031554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.031532    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"default-dockercfg-md425\""
Jul 24 13:53:50.031655 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.031541    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"kube-root-ca.crt\""
Jul 24 13:53:50.031833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.031818    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0a0df7\"/\"rate-test-model-0a0df7-kserve-self-signed-certs\""
Jul 24 13:53:50.038084 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.038062    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:53:50.124797 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.124759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.125002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.124830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.125002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.124858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.125002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.124953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.125002 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.124989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.125184 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.125016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpd9\" (UniqueName: \"kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.225971    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226023 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226033    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226140    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226224    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpd9\" (UniqueName: \"kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226498 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226481    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226552 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.226599 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.226552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.228316 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.228291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.228795 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.228772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.233998 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.233974    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpd9\" (UniqueName: \"kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9\") pod \"rate-test-model-0a0df7-kserve-548bc7895-gdsfx\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") " pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.338384 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.338309    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:53:50.485359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:50.480053    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:53:51.033437 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:51.033397    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" event={"ID":"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7","Type":"ContainerStarted","Data":"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"}
Jul 24 13:53:51.033829 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:51.033445    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" event={"ID":"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7","Type":"ContainerStarted","Data":"18173265ffc731ef296cd2fe2256be494bb44b22cabaef39103de57db45419af"}
Jul 24 13:53:51.049281 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:53:51.049232    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" podStartSLOduration=1.049218031 podStartE2EDuration="1.049218031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:53:51.047826317 +0000 UTC m=+2650.272309643" watchObservedRunningTime="2026-07-24 13:53:51.049218031 +0000 UTC m=+2650.273701362"
Jul 24 13:54:00.338751 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:00.338719    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:00.339318 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:00.338777    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:00.350734 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:00.350713    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:01.094455 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:01.094427    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:33.991947 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:33.991845    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:54:34.944236 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:34.944204    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:54:41.373975 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:41.373947    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:54:41.380721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:41.380697    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:54:42.131730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:42.131700    2556 scope.go:117] "RemoveContainer" containerID="2be1fcd7cf3e337cb294be470f5175f532968633eb25ced9c9d191e1bcd4cbec"
Jul 24 13:54:49.011252 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.011190    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" podUID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" containerName="main" containerID="cri-o://07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe" gracePeriod=45
Jul 24 13:54:49.258378 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.258350    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:54:49.314248 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.314216    2556 generic.go:356] "Generic (PLEG): container finished" podID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" containerID="07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe" exitCode=0
Jul 24 13:54:49.314433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.314265    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"
Jul 24 13:54:49.314433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.314302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" event={"ID":"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d","Type":"ContainerDied","Data":"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"}
Jul 24 13:54:49.314433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.314348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc" event={"ID":"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d","Type":"ContainerDied","Data":"0c3745c15898f29b3da00fe161eeb45c8a4f201581a8dfb65425d29fc7cc3368"}
Jul 24 13:54:49.314433 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.314374    2556 scope.go:117] "RemoveContainer" containerID="07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"
Jul 24 13:54:49.322809 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.322791    2556 scope.go:117] "RemoveContainer" containerID="07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"
Jul 24 13:54:49.323166 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:54:49.323147    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe\": container with ID starting with 07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe not found: ID does not exist" containerID="07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"
Jul 24 13:54:49.323232 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.323175    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe"} err="failed to get container status \"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe\": rpc error: code = NotFound desc = could not find container \"07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe\": container with ID starting with 07a33cece76f7b45eef85fe93843fa538643fadec59006365c5c0949ad7818fe not found: ID does not exist"
Jul 24 13:54:49.378554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378525    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr788\" (UniqueName: \"kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.378712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378569    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.378712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.378712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378608    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.378712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378670    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.378712 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378694    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home\") pod \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\" (UID: \"aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d\") "
Jul 24 13:54:49.379020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378846    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:49.379020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.378989    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:49.379101 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.379026    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home" (OuterVolumeSpecName: "home") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:49.379146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.379103    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.379146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.379120    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.379146 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.379133    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.380872 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.380843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm" (OuterVolumeSpecName: "dshm") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:49.381001 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.380921    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:54:49.381001 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.380991    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788" (OuterVolumeSpecName: "kube-api-access-tr788") pod "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" (UID: "aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d"). InnerVolumeSpecName "kube-api-access-tr788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:54:49.480280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.480247    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr788\" (UniqueName: \"kubernetes.io/projected/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-kube-api-access-tr788\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.480280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.480272    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.480280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.480284    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:49.641873 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.641835    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:54:49.645019 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.644992    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-934481/rate-test-model-934481-kserve-945ddd879-kqtcc"]
Jul 24 13:54:49.963964 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:49.963849    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" podUID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" containerName="main" containerID="cri-o://d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b" gracePeriod=45
Jul 24 13:54:50.206415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.206390    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:50.286597 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286569    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.286783 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286653    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.286783 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286691    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qpd9\" (UniqueName: \"kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.286783 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286728    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.286783 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.287031 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286811    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir\") pod \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\" (UID: \"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7\") "
Jul 24 13:54:50.287031 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.286979    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:50.287148 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.287028    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home" (OuterVolumeSpecName: "home") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:50.287148 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.287103    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:50.287254 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.287180    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:50.287254 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.287195    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:50.287254 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.287204    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:50.288852 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.288828    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:54:50.288988 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.288917    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm" (OuterVolumeSpecName: "dshm") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:54:50.289038 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.289005    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9" (OuterVolumeSpecName: "kube-api-access-5qpd9") pod "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" (UID: "65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7"). InnerVolumeSpecName "kube-api-access-5qpd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:54:50.322392 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.322355    2556 generic.go:356] "Generic (PLEG): container finished" podID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" containerID="d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b" exitCode=0
Jul 24 13:54:50.322542 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.322402    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"
Jul 24 13:54:50.322542 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.322438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" event={"ID":"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7","Type":"ContainerDied","Data":"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"}
Jul 24 13:54:50.322542 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.322480    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx" event={"ID":"65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7","Type":"ContainerDied","Data":"18173265ffc731ef296cd2fe2256be494bb44b22cabaef39103de57db45419af"}
Jul 24 13:54:50.322542 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.322497    2556 scope.go:117] "RemoveContainer" containerID="d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"
Jul 24 13:54:50.331843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.331827    2556 scope.go:117] "RemoveContainer" containerID="d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"
Jul 24 13:54:50.332123 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:54:50.332099    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b\": container with ID starting with d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b not found: ID does not exist" containerID="d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"
Jul 24 13:54:50.332185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.332131    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b"} err="failed to get container status \"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b\": rpc error: code = NotFound desc = could not find container \"d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b\": container with ID starting with d07cedd59d5715a39b818c366ba496fcf2e64205876bab4f29c30e6e08d12f1b not found: ID does not exist"
Jul 24 13:54:50.344413 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.344391    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:54:50.346068 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.346050    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0a0df7/rate-test-model-0a0df7-kserve-548bc7895-gdsfx"]
Jul 24 13:54:50.388020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.387991    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpd9\" (UniqueName: \"kubernetes.io/projected/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-kube-api-access-5qpd9\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:50.388020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.388014    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:50.388020 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:50.388024    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:54:51.346693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:51.344325    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" path="/var/lib/kubelet/pods/65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7/volumes"
Jul 24 13:54:51.346693 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:51.344970    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" path="/var/lib/kubelet/pods/aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d/volumes"
Jul 24 13:54:58.854763 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.854722    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:54:58.855167 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855139    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" containerName="main"
Jul 24 13:54:58.855167 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855151    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" containerName="main"
Jul 24 13:54:58.855167 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855164    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" containerName="main"
Jul 24 13:54:58.855268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855170    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" containerName="main"
Jul 24 13:54:58.855268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855226    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeeeeb84-4a6c-4020-9c7e-4bbe7a67781d" containerName="main"
Jul 24 13:54:58.855268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.855235    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="65001ef0-de27-4a36-a7aa-8bfe9d4b9fd7" containerName="main"
Jul 24 13:54:58.859399 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.859381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:58.861600 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.861575    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e7750f-serving-cert\""
Jul 24 13:54:58.868353 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.868326    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:54:58.965843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.965799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:58.965843 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.965864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:58.966081 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:58.965995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5r5\" (UniqueName: \"kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.066507 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.066473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.066677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.066518    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.066677 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.066562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5r5\" (UniqueName: \"kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.066677 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:54:59.066617    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-serving-cert: secret "maas-api-e2e-ipp-a-e7750f-serving-cert" not found
Jul 24 13:54:59.066799 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:54:59.066697    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls podName:ec9e1756-41dc-4769-9bd7-3e47a8cd7590 nodeName:}" failed. No retries permitted until 2026-07-24 13:54:59.566679129 +0000 UTC m=+2718.791162447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls") pod "maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" (UID: "ec9e1756-41dc-4769-9bd7-3e47a8cd7590") : secret "maas-api-e2e-ipp-a-e7750f-serving-cert" not found
Jul 24 13:54:59.067254 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.067236    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.077270 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.077246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5r5\" (UniqueName: \"kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.497415 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.497382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:54:59.501071 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.501053    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.503256 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.503238    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e7750f-dockercfg-4km5m\""
Jul 24 13:54:59.503357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.503336    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e7750f\""
Jul 24 13:54:59.513109 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.513090    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:54:59.516506 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.516487    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:54:59.516607 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.516595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.526891 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.526868    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:54:59.571447 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.571417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgn5v\" (UniqueName: \"kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.571572 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.571450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdphw\" (UniqueName: \"kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.571572 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.571531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.571665 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.571590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.571665 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.571627    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.573911 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.573893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.672562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.672507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.672562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.672564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.672839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.672603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgn5v\" (UniqueName: \"kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.672839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.672622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdphw\" (UniqueName: \"kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.673339 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.673314    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.673435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.673321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.681128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.681105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgn5v\" (UniqueName: \"kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v\") pod \"payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.681228 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.681193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdphw\" (UniqueName: \"kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw\") pod \"payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.770282 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.770195    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:54:59.811316 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.811285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:54:59.827855 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.827752    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:54:59.945639 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.945610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:54:59.947582 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:54:59.947547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9e1756_41dc_4769_9bd7_3e47a8cd7590.slice/crio-697ab5d8909c268d69fb775389829e02e9d5ce6f7bf5c51694e4ecdcf910ac68 WatchSource:0}: Error finding container 697ab5d8909c268d69fb775389829e02e9d5ce6f7bf5c51694e4ecdcf910ac68: Status 404 returned error can't find the container with id 697ab5d8909c268d69fb775389829e02e9d5ce6f7bf5c51694e4ecdcf910ac68
Jul 24 13:54:59.949280 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.949223    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:54:59.981769 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:54:59.980804    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:54:59.983053 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:54:59.983007    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9eb6ac_1ff8_4ec2_83f0_d516b647c1be.slice/crio-faaac870b9f0ec42521f4db9b9ce4a6863064d817acbaa1ba14e7ad83fd69deb WatchSource:0}: Error finding container faaac870b9f0ec42521f4db9b9ce4a6863064d817acbaa1ba14e7ad83fd69deb: Status 404 returned error can't find the container with id faaac870b9f0ec42521f4db9b9ce4a6863064d817acbaa1ba14e7ad83fd69deb
Jul 24 13:55:00.006911 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.006841    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:55:00.027126 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:55:00.027096    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455c56a5_121f_46a3_a02e_e5f29565b562.slice/crio-70cef3de1e9f89b35548b3bc74710d9936e06e5f117578dd6a00c28e074221f9 WatchSource:0}: Error finding container 70cef3de1e9f89b35548b3bc74710d9936e06e5f117578dd6a00c28e074221f9: Status 404 returned error can't find the container with id 70cef3de1e9f89b35548b3bc74710d9936e06e5f117578dd6a00c28e074221f9
Jul 24 13:55:00.376266 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.376229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" event={"ID":"455c56a5-121f-46a3-a02e-e5f29565b562","Type":"ContainerStarted","Data":"7c60359ae273f1f8d76726e5be0f3e206203d1ed1a26f6364586ab1eaac867ce"}
Jul 24 13:55:00.376398 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.376374    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:55:00.376475 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.376436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" event={"ID":"455c56a5-121f-46a3-a02e-e5f29565b562","Type":"ContainerStarted","Data":"70cef3de1e9f89b35548b3bc74710d9936e06e5f117578dd6a00c28e074221f9"}
Jul 24 13:55:00.377689 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.377664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" event={"ID":"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be","Type":"ContainerStarted","Data":"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"}
Jul 24 13:55:00.377800 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.377704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" event={"ID":"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be","Type":"ContainerStarted","Data":"faaac870b9f0ec42521f4db9b9ce4a6863064d817acbaa1ba14e7ad83fd69deb"}
Jul 24 13:55:00.377800 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.377766    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:55:00.378777 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.378759    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" event={"ID":"ec9e1756-41dc-4769-9bd7-3e47a8cd7590","Type":"ContainerStarted","Data":"697ab5d8909c268d69fb775389829e02e9d5ce6f7bf5c51694e4ecdcf910ac68"}
Jul 24 13:55:00.392664 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.392619    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" podStartSLOduration=1.392603512 podStartE2EDuration="1.392603512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:55:00.391512444 +0000 UTC m=+2719.615995772" watchObservedRunningTime="2026-07-24 13:55:00.392603512 +0000 UTC m=+2719.617086840"
Jul 24 13:55:00.408331 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:00.408289    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" podStartSLOduration=1.408275374 podStartE2EDuration="1.408275374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:55:00.406408043 +0000 UTC m=+2719.630891371" watchObservedRunningTime="2026-07-24 13:55:00.408275374 +0000 UTC m=+2719.632758700"
Jul 24 13:55:01.385650 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:01.385616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" event={"ID":"ec9e1756-41dc-4769-9bd7-3e47a8cd7590","Type":"ContainerStarted","Data":"f11c6cee499daf18c4c38ce5001a0fd8c6d2060ba24eb8b120bdf04dc50610b1"}
Jul 24 13:55:01.386136 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:01.386060    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:55:01.407562 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:01.407502    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" podStartSLOduration=2.98714424 podStartE2EDuration="3.407487454s" totalImagesPullingTime="420.343214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:55:01.406252853 +0000 UTC m=+2720.630736180" watchObservedRunningTime="2026-07-24 13:55:01.407487454 +0000 UTC m=+2720.631970780"
Jul 24 13:55:08.401483 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:08.401450    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:55:11.388372 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:11.388339    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:55:11.388775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:55:11.388411    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:56:07.932766 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.932686    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:56:07.936252 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.936235    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:07.939234 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.939208    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e7750f\"/\"kube-root-ca.crt\""
Jul 24 13:56:07.939362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.939243    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e7750f\"/\"default-dockercfg-ht7s4\""
Jul 24 13:56:07.939362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.939253    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e7750f\"/\"ipp-route-e7750f-kserve-self-signed-certs\""
Jul 24 13:56:07.939362 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.939295    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e7750f\"/\"openshift-service-ca.crt\""
Jul 24 13:56:07.944730 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:07.944707    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:56:08.014174 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.014357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.014357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.014357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.014357 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhmv\" (UniqueName: \"kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.014540 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.014402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115619 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115577    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115619 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhmv\" (UniqueName: \"kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.115851 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.115740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.116173 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.116152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.116232 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.116157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.116232 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.116190    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.118079 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.118058    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.118277 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.118259    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.122704 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.122683    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhmv\" (UniqueName: \"kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv\") pod \"ipp-route-e7750f-kserve-974cb457f-vzrmz\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") " pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.246621 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.246527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:08.380207 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.380179    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:56:08.382584 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:56:08.382552    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a5dd3f1_b400_4c0b_b2ab_846dbbbe9b1b.slice/crio-9da33a5718dfd7ff5e72da4fce9fbea26b8a155030448e005c518afa3145ac76 WatchSource:0}: Error finding container 9da33a5718dfd7ff5e72da4fce9fbea26b8a155030448e005c518afa3145ac76: Status 404 returned error can't find the container with id 9da33a5718dfd7ff5e72da4fce9fbea26b8a155030448e005c518afa3145ac76
Jul 24 13:56:08.715580 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.715547    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" event={"ID":"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b","Type":"ContainerStarted","Data":"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"}
Jul 24 13:56:08.715580 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.715589    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" event={"ID":"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b","Type":"ContainerStarted","Data":"9da33a5718dfd7ff5e72da4fce9fbea26b8a155030448e005c518afa3145ac76"}
Jul 24 13:56:08.731503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:08.731451    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" podStartSLOduration=1.731436845 podStartE2EDuration="1.731436845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:56:08.730745425 +0000 UTC m=+2787.955228754" watchObservedRunningTime="2026-07-24 13:56:08.731436845 +0000 UTC m=+2787.955920170"
Jul 24 13:56:18.246953 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:18.246888    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:18.249388 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:18.246989    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:18.260049 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:18.260022    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:18.775048 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:18.775014    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:56:39.416022 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:39.415989    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr_455c56a5-121f-46a3-a02e-e5f29565b562/payload-processing/0.log"
Jul 24 13:56:44.042803 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.042760    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:56:44.046680 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.046657    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.048903 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.048882    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-095476-dockercfg-kkzb4\""
Jul 24 13:56:44.049150 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.049134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-095476\""
Jul 24 13:56:44.052758 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.052739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.052869 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.052815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5klv\" (UniqueName: \"kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.061375 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.061352    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:56:44.153606 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.153572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5klv\" (UniqueName: \"kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.153789 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.153625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.154354 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.154333    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.163078 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.163052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5klv\" (UniqueName: \"kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv\") pod \"payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.356716 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.356628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.705765 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.705612    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:56:44.708561 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:56:44.708534    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3465d2_e666_41e2_bd9e_4bdafd4efb46.slice/crio-f18d225397228702137ec1627cef192b3f5779d9d88a3612f410db26cb97ff0c WatchSource:0}: Error finding container f18d225397228702137ec1627cef192b3f5779d9d88a3612f410db26cb97ff0c: Status 404 returned error can't find the container with id f18d225397228702137ec1627cef192b3f5779d9d88a3612f410db26cb97ff0c
Jul 24 13:56:44.897467 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.897435    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" event={"ID":"ae3465d2-e666-41e2-bd9e-4bdafd4efb46","Type":"ContainerStarted","Data":"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"}
Jul 24 13:56:44.897650 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.897485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" event={"ID":"ae3465d2-e666-41e2-bd9e-4bdafd4efb46","Type":"ContainerStarted","Data":"f18d225397228702137ec1627cef192b3f5779d9d88a3612f410db26cb97ff0c"}
Jul 24 13:56:44.897650 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.897546    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:56:44.916355 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:44.916309    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" podStartSLOduration=0.916292501 podStartE2EDuration="916.292501ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:56:44.914886163 +0000 UTC m=+2824.139369492" watchObservedRunningTime="2026-07-24 13:56:44.916292501 +0000 UTC m=+2824.140775827"
Jul 24 13:56:55.904053 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:56:55.904021    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:57:09.015759 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.015723    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"]
Jul 24 13:57:09.018822 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.018802    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:09.020866 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.020849    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:57:09.026833 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.026811    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"]
Jul 24 13:57:09.082418 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.082385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w\" (UID: \"9d236c39-bd41-4e05-80a8-179df71a39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:09.183581 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.183547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w\" (UID: \"9d236c39-bd41-4e05-80a8-179df71a39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:09.184188 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.184164    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w\" (UID: \"9d236c39-bd41-4e05-80a8-179df71a39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:09.329641 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.329603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:09.461536 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:09.461421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"]
Jul 24 13:57:09.464893 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:57:09.464862    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d236c39_bd41_4e05_80a8_179df71a39b0.slice/crio-51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033 WatchSource:0}: Error finding container 51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033: Status 404 returned error can't find the container with id 51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033
Jul 24 13:57:10.025629 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:10.025594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w" event={"ID":"9d236c39-bd41-4e05-80a8-179df71a39b0","Type":"ContainerStarted","Data":"b931a643ef9e15e9db65aaecd5702d276f398cf627f53266ff5096f94aae71d1"}
Jul 24 13:57:10.026131 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:10.025654    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w" event={"ID":"9d236c39-bd41-4e05-80a8-179df71a39b0","Type":"ContainerStarted","Data":"51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033"}
Jul 24 13:57:10.041944 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:10.041884    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w" podStartSLOduration=1.041870726 podStartE2EDuration="1.041870726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:57:10.039047631 +0000 UTC m=+2849.263530958" watchObservedRunningTime="2026-07-24 13:57:10.041870726 +0000 UTC m=+2849.266354054"
Jul 24 13:57:11.032439 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:11.032409    2556 generic.go:356] "Generic (PLEG): container finished" podID="9d236c39-bd41-4e05-80a8-179df71a39b0" containerID="b931a643ef9e15e9db65aaecd5702d276f398cf627f53266ff5096f94aae71d1" exitCode=0
Jul 24 13:57:11.032857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:11.032506    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w" event={"ID":"9d236c39-bd41-4e05-80a8-179df71a39b0","Type":"ContainerDied","Data":"b931a643ef9e15e9db65aaecd5702d276f398cf627f53266ff5096f94aae71d1"}
Jul 24 13:57:12.172192 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:12.172167    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:12.316252 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:12.316168    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca\") pod \"9d236c39-bd41-4e05-80a8-179df71a39b0\" (UID: \"9d236c39-bd41-4e05-80a8-179df71a39b0\") "
Jul 24 13:57:12.316477 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:12.316454    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d236c39-bd41-4e05-80a8-179df71a39b0" (UID: "9d236c39-bd41-4e05-80a8-179df71a39b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:12.417452 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:12.417420    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d236c39-bd41-4e05-80a8-179df71a39b0-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:13.043491 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:13.043452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w" event={"ID":"9d236c39-bd41-4e05-80a8-179df71a39b0","Type":"ContainerDied","Data":"51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033"}
Jul 24 13:57:13.043671 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:13.043535    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a7d398b0ac98140f6eac48e278d46b4de56ef3350674b3cb17ecfb385a4033"
Jul 24 13:57:13.043671 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:13.043460    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"
Jul 24 13:57:19.360185 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.360154    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:57:19.360663 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.360359    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" podUID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" containerName="payload-processing" containerID="cri-o://bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568" gracePeriod=30
Jul 24 13:57:19.613289 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.613230    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:57:19.788542 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.788510    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume\") pod \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") "
Jul 24 13:57:19.788724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.788600    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5klv\" (UniqueName: \"kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv\") pod \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\" (UID: \"ae3465d2-e666-41e2-bd9e-4bdafd4efb46\") "
Jul 24 13:57:19.788899 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.788873    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae3465d2-e666-41e2-bd9e-4bdafd4efb46" (UID: "ae3465d2-e666-41e2-bd9e-4bdafd4efb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:19.790778 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.790755    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv" (OuterVolumeSpecName: "kube-api-access-j5klv") pod "ae3465d2-e666-41e2-bd9e-4bdafd4efb46" (UID: "ae3465d2-e666-41e2-bd9e-4bdafd4efb46"). InnerVolumeSpecName "kube-api-access-j5klv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:19.889951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.889845    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:19.889951 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:19.889873    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5klv\" (UniqueName: \"kubernetes.io/projected/ae3465d2-e666-41e2-bd9e-4bdafd4efb46-kube-api-access-j5klv\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:20.080649 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.080613    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" containerID="bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568" exitCode=0
Jul 24 13:57:20.080839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.080666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" event={"ID":"ae3465d2-e666-41e2-bd9e-4bdafd4efb46","Type":"ContainerDied","Data":"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"}
Jul 24 13:57:20.080839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.080731    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l" event={"ID":"ae3465d2-e666-41e2-bd9e-4bdafd4efb46","Type":"ContainerDied","Data":"f18d225397228702137ec1627cef192b3f5779d9d88a3612f410db26cb97ff0c"}
Jul 24 13:57:20.080839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.080668    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"
Jul 24 13:57:20.080839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.080758    2556 scope.go:117] "RemoveContainer" containerID="bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"
Jul 24 13:57:20.090128 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.090101    2556 scope.go:117] "RemoveContainer" containerID="bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"
Jul 24 13:57:20.090366 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:57:20.090344    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568\": container with ID starting with bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568 not found: ID does not exist" containerID="bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"
Jul 24 13:57:20.090421 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.090381    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568"} err="failed to get container status \"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568\": rpc error: code = NotFound desc = could not find container \"bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568\": container with ID starting with bbab65e1629c2ee60d06ab8bcf6b5e0044284d1929c3d18aca6d1626ec4cf568 not found: ID does not exist"
Jul 24 13:57:20.101740 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.101719    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:57:20.104359 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:20.104335    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-095476-6cfd4b7b66-6hz7l"]
Jul 24 13:57:21.341470 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:21.341438    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" path="/var/lib/kubelet/pods/ae3465d2-e666-41e2-bd9e-4bdafd4efb46/volumes"
Jul 24 13:57:24.093551 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:24.093518    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"]
Jul 24 13:57:24.095453 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:24.095427    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-095476-njb4w"]
Jul 24 13:57:25.341257 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:25.341218    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d236c39-bd41-4e05-80a8-179df71a39b0" path="/var/lib/kubelet/pods/9d236c39-bd41-4e05-80a8-179df71a39b0/volumes"
Jul 24 13:57:35.148310 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148219    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"]
Jul 24 13:57:35.148841 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148822    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d236c39-bd41-4e05-80a8-179df71a39b0" containerName="revoke-keys"
Jul 24 13:57:35.148924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148844    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d236c39-bd41-4e05-80a8-179df71a39b0" containerName="revoke-keys"
Jul 24 13:57:35.148924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" containerName="payload-processing"
Jul 24 13:57:35.148924 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148879    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" containerName="payload-processing"
Jul 24 13:57:35.149107 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.148996    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3465d2-e666-41e2-bd9e-4bdafd4efb46" containerName="payload-processing"
Jul 24 13:57:35.149107 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.149014    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d236c39-bd41-4e05-80a8-179df71a39b0" containerName="revoke-keys"
Jul 24 13:57:35.153804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.153780    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:35.156771 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.156750    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:57:35.169788 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.169764    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"]
Jul 24 13:57:35.230867 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.230832    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4\" (UID: \"2bba6b8d-46a3-42b1-ba71-c3ac192df837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:35.332025 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.331988    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4\" (UID: \"2bba6b8d-46a3-42b1-ba71-c3ac192df837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:35.332613 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.332596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4\" (UID: \"2bba6b8d-46a3-42b1-ba71-c3ac192df837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:35.464642 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.464554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:35.822666 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:35.822638    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"]
Jul 24 13:57:35.824591 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:57:35.824562    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bba6b8d_46a3_42b1_ba71_c3ac192df837.slice/crio-2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697 WatchSource:0}: Error finding container 2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697: Status 404 returned error can't find the container with id 2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697
Jul 24 13:57:36.161419 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:36.161384    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4" event={"ID":"2bba6b8d-46a3-42b1-ba71-c3ac192df837","Type":"ContainerStarted","Data":"440681c721415eca5a85e8d7bdd08237ef09f707f2358557384de74ee00c2ff9"}
Jul 24 13:57:36.161787 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:36.161442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4" event={"ID":"2bba6b8d-46a3-42b1-ba71-c3ac192df837","Type":"ContainerStarted","Data":"2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697"}
Jul 24 13:57:36.180707 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:36.180661    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4" podStartSLOduration=1.180646138 podStartE2EDuration="1.180646138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:57:36.180037541 +0000 UTC m=+2875.404520870" watchObservedRunningTime="2026-07-24 13:57:36.180646138 +0000 UTC m=+2875.405129509"
Jul 24 13:57:37.169918 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:37.169884    2556 generic.go:356] "Generic (PLEG): container finished" podID="2bba6b8d-46a3-42b1-ba71-c3ac192df837" containerID="440681c721415eca5a85e8d7bdd08237ef09f707f2358557384de74ee00c2ff9" exitCode=0
Jul 24 13:57:37.170321 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:37.169974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4" event={"ID":"2bba6b8d-46a3-42b1-ba71-c3ac192df837","Type":"ContainerDied","Data":"440681c721415eca5a85e8d7bdd08237ef09f707f2358557384de74ee00c2ff9"}
Jul 24 13:57:38.306514 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:38.306488    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:38.360697 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:38.360666    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca\") pod \"2bba6b8d-46a3-42b1-ba71-c3ac192df837\" (UID: \"2bba6b8d-46a3-42b1-ba71-c3ac192df837\") "
Jul 24 13:57:38.361100 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:38.361056    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bba6b8d-46a3-42b1-ba71-c3ac192df837" (UID: "2bba6b8d-46a3-42b1-ba71-c3ac192df837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:38.462015 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:38.461924    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bba6b8d-46a3-42b1-ba71-c3ac192df837-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:39.182361 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:39.182327    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"
Jul 24 13:57:39.182537 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:39.182358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4" event={"ID":"2bba6b8d-46a3-42b1-ba71-c3ac192df837","Type":"ContainerDied","Data":"2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697"}
Jul 24 13:57:39.182537 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:39.182405    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f89b690fa4de6bf6a6f151ad0ac5d694909b4926239cbdd3979ece5c47b4697"
Jul 24 13:57:51.064797 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.064763    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:57:51.065221 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.065015    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" podUID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" containerName="maas-api" containerID="cri-o://f11c6cee499daf18c4c38ce5001a0fd8c6d2060ba24eb8b120bdf04dc50610b1" gracePeriod=30
Jul 24 13:57:51.124713 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.124677    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:57:51.125010 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.124959    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" containerName="payload-processing" containerID="cri-o://7c60359ae273f1f8d76726e5be0f3e206203d1ed1a26f6364586ab1eaac867ce" gracePeriod=30
Jul 24 13:57:51.136187 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.136161    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:57:51.246776 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.246745    2556 generic.go:356] "Generic (PLEG): container finished" podID="455c56a5-121f-46a3-a02e-e5f29565b562" containerID="7c60359ae273f1f8d76726e5be0f3e206203d1ed1a26f6364586ab1eaac867ce" exitCode=0
Jul 24 13:57:51.246890 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.246836    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" event={"ID":"455c56a5-121f-46a3-a02e-e5f29565b562","Type":"ContainerDied","Data":"7c60359ae273f1f8d76726e5be0f3e206203d1ed1a26f6364586ab1eaac867ce"}
Jul 24 13:57:51.248667 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.248643    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" containerID="f11c6cee499daf18c4c38ce5001a0fd8c6d2060ba24eb8b120bdf04dc50610b1" exitCode=0
Jul 24 13:57:51.248766 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.248680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" event={"ID":"ec9e1756-41dc-4769-9bd7-3e47a8cd7590","Type":"ContainerDied","Data":"f11c6cee499daf18c4c38ce5001a0fd8c6d2060ba24eb8b120bdf04dc50610b1"}
Jul 24 13:57:51.249008 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.248985    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerName="payload-pre-processing" containerID="cri-o://bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9" gracePeriod=30
Jul 24 13:57:51.377247 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.377225    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:57:51.386112 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.386069    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.73:9004: connect: connection refused"
Jul 24 13:57:51.404610 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.404586    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:57:51.482246 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482217    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgn5v\" (UniqueName: \"kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v\") pod \"455c56a5-121f-46a3-a02e-e5f29565b562\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") "
Jul 24 13:57:51.482435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca\") pod \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") "
Jul 24 13:57:51.482435 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482277    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume\") pod \"455c56a5-121f-46a3-a02e-e5f29565b562\" (UID: \"455c56a5-121f-46a3-a02e-e5f29565b562\") "
Jul 24 13:57:51.482554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482468    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk5r5\" (UniqueName: \"kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5\") pod \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") "
Jul 24 13:57:51.482554 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482510    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") pod \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\" (UID: \"ec9e1756-41dc-4769-9bd7-3e47a8cd7590\") "
Jul 24 13:57:51.482838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482662    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec9e1756-41dc-4769-9bd7-3e47a8cd7590" (UID: "ec9e1756-41dc-4769-9bd7-3e47a8cd7590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:51.482838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482685    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "455c56a5-121f-46a3-a02e-e5f29565b562" (UID: "455c56a5-121f-46a3-a02e-e5f29565b562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:51.482916 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482867    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.482916 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.482887    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455c56a5-121f-46a3-a02e-e5f29565b562-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.484753 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.484726    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec9e1756-41dc-4769-9bd7-3e47a8cd7590" (UID: "ec9e1756-41dc-4769-9bd7-3e47a8cd7590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:57:51.484863 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.484845    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5" (OuterVolumeSpecName: "kube-api-access-mk5r5") pod "ec9e1756-41dc-4769-9bd7-3e47a8cd7590" (UID: "ec9e1756-41dc-4769-9bd7-3e47a8cd7590"). InnerVolumeSpecName "kube-api-access-mk5r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:51.484904 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.484878    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v" (OuterVolumeSpecName: "kube-api-access-fgn5v") pod "455c56a5-121f-46a3-a02e-e5f29565b562" (UID: "455c56a5-121f-46a3-a02e-e5f29565b562"). InnerVolumeSpecName "kube-api-access-fgn5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:51.500820 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.500799    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:57:51.583775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.583684    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume\") pod \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") "
Jul 24 13:57:51.583979 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.583784    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdphw\" (UniqueName: \"kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw\") pod \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\" (UID: \"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be\") "
Jul 24 13:57:51.584062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.584038    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgn5v\" (UniqueName: \"kubernetes.io/projected/455c56a5-121f-46a3-a02e-e5f29565b562-kube-api-access-fgn5v\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.584062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.584053    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk5r5\" (UniqueName: \"kubernetes.io/projected/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-kube-api-access-mk5r5\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.584062 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.584052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" (UID: "1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:57:51.584210 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.584062    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9e1756-41dc-4769-9bd7-3e47a8cd7590-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.585982 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.585960    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw" (OuterVolumeSpecName: "kube-api-access-gdphw") pod "1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" (UID: "1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be"). InnerVolumeSpecName "kube-api-access-gdphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:51.685060 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.685024    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:51.685060 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:51.685052    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdphw\" (UniqueName: \"kubernetes.io/projected/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be-kube-api-access-gdphw\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:57:52.255163 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.255126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" event={"ID":"455c56a5-121f-46a3-a02e-e5f29565b562","Type":"ContainerDied","Data":"70cef3de1e9f89b35548b3bc74710d9936e06e5f117578dd6a00c28e074221f9"}
Jul 24 13:57:52.255593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.255142    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"
Jul 24 13:57:52.255593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.255233    2556 scope.go:117] "RemoveContainer" containerID="7c60359ae273f1f8d76726e5be0f3e206203d1ed1a26f6364586ab1eaac867ce"
Jul 24 13:57:52.256615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.256479    2556 generic.go:356] "Generic (PLEG): container finished" podID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerID="bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9" exitCode=0
Jul 24 13:57:52.256615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.256538    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"
Jul 24 13:57:52.256615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.256586    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" event={"ID":"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be","Type":"ContainerDied","Data":"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"}
Jul 24 13:57:52.256775 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.256626    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst" event={"ID":"1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be","Type":"ContainerDied","Data":"faaac870b9f0ec42521f4db9b9ce4a6863064d817acbaa1ba14e7ad83fd69deb"}
Jul 24 13:57:52.258370 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.258343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp" event={"ID":"ec9e1756-41dc-4769-9bd7-3e47a8cd7590","Type":"ContainerDied","Data":"697ab5d8909c268d69fb775389829e02e9d5ce6f7bf5c51694e4ecdcf910ac68"}
Jul 24 13:57:52.258485 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.258377    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"
Jul 24 13:57:52.266337 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.266244    2556 scope.go:117] "RemoveContainer" containerID="bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"
Jul 24 13:57:52.274416 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.274401    2556 scope.go:117] "RemoveContainer" containerID="bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"
Jul 24 13:57:52.274649 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:57:52.274632    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9\": container with ID starting with bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9 not found: ID does not exist" containerID="bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"
Jul 24 13:57:52.274694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.274655    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9"} err="failed to get container status \"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9\": rpc error: code = NotFound desc = could not find container \"bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9\": container with ID starting with bb438e5704a17a86236d38067337a138db4de047bf84966e3a8845538bddcdc9 not found: ID does not exist"
Jul 24 13:57:52.274694 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.274670    2556 scope.go:117] "RemoveContainer" containerID="f11c6cee499daf18c4c38ce5001a0fd8c6d2060ba24eb8b120bdf04dc50610b1"
Jul 24 13:57:52.280284 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.280259    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:57:52.282294 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.282273    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e7750f-6f8f6c6956-k9bst"]
Jul 24 13:57:52.295669 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.295645    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:57:52.299544 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.299521    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e7750f-6b986b5c5f-2nlgp"]
Jul 24 13:57:52.307879 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.307858    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:57:52.311687 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.311667    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr"]
Jul 24 13:57:52.386401 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:52.386327    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-e7750f-76f9fd6598-gpghr" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.74:9004: i/o timeout"
Jul 24 13:57:53.343440 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:53.343400    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" path="/var/lib/kubelet/pods/1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be/volumes"
Jul 24 13:57:53.343961 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:53.343922    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" path="/var/lib/kubelet/pods/455c56a5-121f-46a3-a02e-e5f29565b562/volumes"
Jul 24 13:57:53.344368 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:53.344350    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" path="/var/lib/kubelet/pods/ec9e1756-41dc-4769-9bd7-3e47a8cd7590/volumes"
Jul 24 13:57:55.208024 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:55.207981    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"]
Jul 24 13:57:55.210157 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:55.210134    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e7750f-qxhw4"]
Jul 24 13:57:55.341138 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:57:55.341106    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bba6b8d-46a3-42b1-ba71-c3ac192df837" path="/var/lib/kubelet/pods/2bba6b8d-46a3-42b1-ba71-c3ac192df837/volumes"
Jul 24 13:58:00.609852 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:00.609814    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:58:15.629839 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.629771    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" podUID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" containerName="main" containerID="cri-o://9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf" gracePeriod=45
Jul 24 13:58:15.875457 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.875433    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:58:15.909550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909469    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrhmv\" (UniqueName: \"kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909519    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909550 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909551    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909582    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909619    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909668    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs\") pod \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\" (UID: \"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b\") "
Jul 24 13:58:15.909849 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.909828    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home" (OuterVolumeSpecName: "home") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:58:15.910118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.910002    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:58:15.910118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.910086    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-model-cache\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:15.910118 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.910111    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-home\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:15.910290 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.910165    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:58:15.911892 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.911849    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm" (OuterVolumeSpecName: "dshm") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:58:15.912036 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.911956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv" (OuterVolumeSpecName: "kube-api-access-vrhmv") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "kube-api-access-vrhmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:58:15.912239 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:15.912223    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" (UID: "8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:58:16.010602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.010565    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhmv\" (UniqueName: \"kubernetes.io/projected/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-kube-api-access-vrhmv\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:16.010602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.010598    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-dshm\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:16.010602 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.010607    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tmp-dir\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:16.010838 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.010615    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b-tls-certs\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:58:16.376171 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.376139    2556 generic.go:356] "Generic (PLEG): container finished" podID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" containerID="9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf" exitCode=0
Jul 24 13:58:16.376377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.376189    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"
Jul 24 13:58:16.376377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.376231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" event={"ID":"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b","Type":"ContainerDied","Data":"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"}
Jul 24 13:58:16.376377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.376272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz" event={"ID":"8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b","Type":"ContainerDied","Data":"9da33a5718dfd7ff5e72da4fce9fbea26b8a155030448e005c518afa3145ac76"}
Jul 24 13:58:16.376377 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.376289    2556 scope.go:117] "RemoveContainer" containerID="9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"
Jul 24 13:58:16.385593 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.385571    2556 scope.go:117] "RemoveContainer" containerID="9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"
Jul 24 13:58:16.385841 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:58:16.385823    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf\": container with ID starting with 9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf not found: ID does not exist" containerID="9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"
Jul 24 13:58:16.385880 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.385850    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf"} err="failed to get container status \"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf\": rpc error: code = NotFound desc = could not find container \"9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf\": container with ID starting with 9a1788f2490cd98146cd4fbc723bcbb98d32cac4a4f303bf81c71989be37cfcf not found: ID does not exist"
Jul 24 13:58:16.398030 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.398005    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:58:16.401984 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:16.401965    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e7750f/ipp-route-e7750f-kserve-974cb457f-vzrmz"]
Jul 24 13:58:17.340991 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:17.340957    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" path="/var/lib/kubelet/pods/8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b/volumes"
Jul 24 13:58:57.838315 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838277    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"]
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838637    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerName="payload-pre-processing"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838650    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerName="payload-pre-processing"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838668    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" containerName="payload-processing"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838674    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" containerName="payload-processing"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838680    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bba6b8d-46a3-42b1-ba71-c3ac192df837" containerName="revoke-keys"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838685    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bba6b8d-46a3-42b1-ba71-c3ac192df837" containerName="revoke-keys"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838693    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" containerName="main"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838698    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" containerName="main"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838708    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" containerName="maas-api"
Jul 24 13:58:57.838724 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838713    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" containerName="maas-api"
Jul 24 13:58:57.839072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838766    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="455c56a5-121f-46a3-a02e-e5f29565b562" containerName="payload-processing"
Jul 24 13:58:57.839072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838773    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5dd3f1-b400-4c0b-b2ab-846dbbbe9b1b" containerName="main"
Jul 24 13:58:57.839072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838781    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9e1756-41dc-4769-9bd7-3e47a8cd7590" containerName="maas-api"
Jul 24 13:58:57.839072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838788    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9eb6ac-1ff8-4ec2-83f0-d516b647c1be" containerName="payload-pre-processing"
Jul 24 13:58:57.839072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.838795    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bba6b8d-46a3-42b1-ba71-c3ac192df837" containerName="revoke-keys"
Jul 24 13:58:57.841859 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.841843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:58:57.844154 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.844132    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:58:57.850648 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.850623    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"]
Jul 24 13:58:57.995112 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:57.995075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh\" (UID: \"83a2d1ec-2180-4dea-9ca9-43285dcc6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:58:58.095785 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:58.095695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh\" (UID: \"83a2d1ec-2180-4dea-9ca9-43285dcc6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:58:58.096322 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:58.096304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh\" (UID: \"83a2d1ec-2180-4dea-9ca9-43285dcc6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:58:58.153313 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:58.153280    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:58:58.290121 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:58.290074    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"]
Jul 24 13:58:58.293336 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:58:58.293311    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a2d1ec_2180_4dea_9ca9_43285dcc6172.slice/crio-5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab WatchSource:0}: Error finding container 5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab: Status 404 returned error can't find the container with id 5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab
Jul 24 13:58:58.584645 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:58.584608    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh" event={"ID":"83a2d1ec-2180-4dea-9ca9-43285dcc6172","Type":"ContainerStarted","Data":"5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab"}
Jul 24 13:58:59.591266 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:59.591230    2556 generic.go:356] "Generic (PLEG): container finished" podID="83a2d1ec-2180-4dea-9ca9-43285dcc6172" containerID="4171408b7bd3a3c32cd299f1ac4b30304bf70f5d66898743c35d722a0e8c5a63" exitCode=0
Jul 24 13:58:59.591685 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:58:59.591281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh" event={"ID":"83a2d1ec-2180-4dea-9ca9-43285dcc6172","Type":"ContainerDied","Data":"4171408b7bd3a3c32cd299f1ac4b30304bf70f5d66898743c35d722a0e8c5a63"}
Jul 24 13:59:00.739275 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:00.739241    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:59:00.922582 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:00.922495    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca\") pod \"83a2d1ec-2180-4dea-9ca9-43285dcc6172\" (UID: \"83a2d1ec-2180-4dea-9ca9-43285dcc6172\") "
Jul 24 13:59:00.922834 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:00.922810    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83a2d1ec-2180-4dea-9ca9-43285dcc6172" (UID: "83a2d1ec-2180-4dea-9ca9-43285dcc6172"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:59:01.023155 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:01.023121    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a2d1ec-2180-4dea-9ca9-43285dcc6172-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:01.602326 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:01.602251    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"
Jul 24 13:59:01.602473 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:01.602249    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh" event={"ID":"83a2d1ec-2180-4dea-9ca9-43285dcc6172","Type":"ContainerDied","Data":"5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab"}
Jul 24 13:59:01.602473 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:01.602378    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a47f68ad0a4b7bc2c27e1ce69696ffea55fec1600c38b575c6a2176dc8ab9ab"
Jul 24 13:59:12.917358 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:12.917308    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"]
Jul 24 13:59:12.918953 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:12.918916    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e01298-cqrjh"]
Jul 24 13:59:13.341425 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:13.341394    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a2d1ec-2180-4dea-9ca9-43285dcc6172" path="/var/lib/kubelet/pods/83a2d1ec-2180-4dea-9ca9-43285dcc6172/volumes"
Jul 24 13:59:19.618267 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.618230    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"]
Jul 24 13:59:19.618681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.618591    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a2d1ec-2180-4dea-9ca9-43285dcc6172" containerName="revoke-keys"
Jul 24 13:59:19.618681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.618602    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a2d1ec-2180-4dea-9ca9-43285dcc6172" containerName="revoke-keys"
Jul 24 13:59:19.618681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.618656    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a2d1ec-2180-4dea-9ca9-43285dcc6172" containerName="revoke-keys"
Jul 24 13:59:19.621530 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.621516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:19.625120 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.625097    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 13:59:19.627893 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.627867    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"]
Jul 24 13:59:19.700036 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.700005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-934481-4pftl\" (UID: \"3c1c0736-dd1a-4657-901c-5f87be30f364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:19.801326 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.801290    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-934481-4pftl\" (UID: \"3c1c0736-dd1a-4657-901c-5f87be30f364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:19.801925 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.801904    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-934481-4pftl\" (UID: \"3c1c0736-dd1a-4657-901c-5f87be30f364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:19.934026 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:19.933952    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:20.067759 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:20.067580    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"]
Jul 24 13:59:20.069991 ip-10-0-136-14 kubenswrapper[2556]: W0724 13:59:20.069961    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1c0736_dd1a_4657_901c_5f87be30f364.slice/crio-331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a WatchSource:0}: Error finding container 331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a: Status 404 returned error can't find the container with id 331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a
Jul 24 13:59:20.699777 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:20.699742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl" event={"ID":"3c1c0736-dd1a-4657-901c-5f87be30f364","Type":"ContainerStarted","Data":"47feca2cc0dc9fa3157f2cdaebe5fcc9e4356eaf8e3aeef98c16e69c95609205"}
Jul 24 13:59:20.700204 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:20.699826    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl" event={"ID":"3c1c0736-dd1a-4657-901c-5f87be30f364","Type":"ContainerStarted","Data":"331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a"}
Jul 24 13:59:20.717223 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:20.717175    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl" podStartSLOduration=1.7171587609999999 podStartE2EDuration="1.717158761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:59:20.716278783 +0000 UTC m=+2979.940762111" watchObservedRunningTime="2026-07-24 13:59:20.717158761 +0000 UTC m=+2979.941642092"
Jul 24 13:59:21.706615 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:21.706574    2556 generic.go:356] "Generic (PLEG): container finished" podID="3c1c0736-dd1a-4657-901c-5f87be30f364" containerID="47feca2cc0dc9fa3157f2cdaebe5fcc9e4356eaf8e3aeef98c16e69c95609205" exitCode=0
Jul 24 13:59:21.707063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:21.706660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl" event={"ID":"3c1c0736-dd1a-4657-901c-5f87be30f364","Type":"ContainerDied","Data":"47feca2cc0dc9fa3157f2cdaebe5fcc9e4356eaf8e3aeef98c16e69c95609205"}
Jul 24 13:59:22.841857 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:22.841830    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:22.928261 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:22.928215    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca\") pod \"3c1c0736-dd1a-4657-901c-5f87be30f364\" (UID: \"3c1c0736-dd1a-4657-901c-5f87be30f364\") "
Jul 24 13:59:22.928571 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:22.928547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c1c0736-dd1a-4657-901c-5f87be30f364" (UID: "3c1c0736-dd1a-4657-901c-5f87be30f364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:59:23.029732 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:23.029703    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c0736-dd1a-4657-901c-5f87be30f364-maas-api-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:23.717391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:23.717311    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"
Jul 24 13:59:23.717391 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:23.717356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl" event={"ID":"3c1c0736-dd1a-4657-901c-5f87be30f364","Type":"ContainerDied","Data":"331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a"}
Jul 24 13:59:23.717585 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:23.717404    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331b95ad2cec50505a4ba381152612cbdb190579e6eb3b69c5dd9e3f8f82cb5a"
Jul 24 13:59:30.165045 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.165005    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:59:30.165466 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.165240    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" podUID="81e9208d-5d19-4add-bebc-deec0dd8e161" containerName="maas-api" containerID="cri-o://286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9" gracePeriod=30
Jul 24 13:59:30.415817 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.415753    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:59:30.501782 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.501745    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca\") pod \"81e9208d-5d19-4add-bebc-deec0dd8e161\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") "
Jul 24 13:59:30.501987 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.501790    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls\") pod \"81e9208d-5d19-4add-bebc-deec0dd8e161\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") "
Jul 24 13:59:30.501987 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.501821    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mt5b\" (UniqueName: \"kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b\") pod \"81e9208d-5d19-4add-bebc-deec0dd8e161\" (UID: \"81e9208d-5d19-4add-bebc-deec0dd8e161\") "
Jul 24 13:59:30.502240 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.502212    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81e9208d-5d19-4add-bebc-deec0dd8e161" (UID: "81e9208d-5d19-4add-bebc-deec0dd8e161"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:59:30.504114 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.504091    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b" (OuterVolumeSpecName: "kube-api-access-6mt5b") pod "81e9208d-5d19-4add-bebc-deec0dd8e161" (UID: "81e9208d-5d19-4add-bebc-deec0dd8e161"). InnerVolumeSpecName "kube-api-access-6mt5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:59:30.504233 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.504199    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81e9208d-5d19-4add-bebc-deec0dd8e161" (UID: "81e9208d-5d19-4add-bebc-deec0dd8e161"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:59:30.602468 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.602432    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e9208d-5d19-4add-bebc-deec0dd8e161-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:30.602468 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.602462    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e9208d-5d19-4add-bebc-deec0dd8e161-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:30.602468 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.602471    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mt5b\" (UniqueName: \"kubernetes.io/projected/81e9208d-5d19-4add-bebc-deec0dd8e161-kube-api-access-6mt5b\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:30.753874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.753776    2556 generic.go:356] "Generic (PLEG): container finished" podID="81e9208d-5d19-4add-bebc-deec0dd8e161" containerID="286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9" exitCode=0
Jul 24 13:59:30.753874 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.753827    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"
Jul 24 13:59:30.754106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.753870    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" event={"ID":"81e9208d-5d19-4add-bebc-deec0dd8e161","Type":"ContainerDied","Data":"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"}
Jul 24 13:59:30.754106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.753925    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz" event={"ID":"81e9208d-5d19-4add-bebc-deec0dd8e161","Type":"ContainerDied","Data":"52b632b72fe78429386e1f0c28f0b4d4b46c72c978b46a4a439edade65351b78"}
Jul 24 13:59:30.754106 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.753954    2556 scope.go:117] "RemoveContainer" containerID="286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"
Jul 24 13:59:30.763973 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.763956    2556 scope.go:117] "RemoveContainer" containerID="286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"
Jul 24 13:59:30.764233 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:59:30.764215    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9\": container with ID starting with 286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9 not found: ID does not exist" containerID="286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"
Jul 24 13:59:30.764299 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.764239    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9"} err="failed to get container status \"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9\": rpc error: code = NotFound desc = could not find container \"286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9\": container with ID starting with 286852e67920a4dc59036255edfc4758a1165678b5da5a0a0dbb46e88ba4d7e9 not found: ID does not exist"
Jul 24 13:59:30.774262 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.774238    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:59:30.777721 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:30.777699    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-934481-699d76cd54-k4bsz"]
Jul 24 13:59:31.341644 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:31.341615    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e9208d-5d19-4add-bebc-deec0dd8e161" path="/var/lib/kubelet/pods/81e9208d-5d19-4add-bebc-deec0dd8e161/volumes"
Jul 24 13:59:34.695290 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:34.695252    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"]
Jul 24 13:59:34.700680 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:34.700655    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-934481-4pftl"]
Jul 24 13:59:35.341373 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:35.341341    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1c0736-dd1a-4657-901c-5f87be30f364" path="/var/lib/kubelet/pods/3c1c0736-dd1a-4657-901c-5f87be30f364/volumes"
Jul 24 13:59:41.406555 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:41.406530    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:59:41.415765 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:41.415744    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7hx9v_2d66e606-fc0c-4031-8314-7320e1ca9c74/ovn-acl-logging/0.log"
Jul 24 13:59:52.380179 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.380146    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:59:52.380758 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.380399    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" podUID="9ab4676e-738e-4812-bc07-3a5a14eab39e" containerName="maas-api" containerID="cri-o://c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac" gracePeriod=30
Jul 24 13:59:52.440268 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.440238    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:59:52.440479 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.440453    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" podUID="600156c9-59d2-4948-8bf2-5b0b045732cc" containerName="payload-processing" containerID="cri-o://8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10" gracePeriod=30
Jul 24 13:59:52.682287 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.682263    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:59:52.713164 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.713133    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:59:52.798003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.797968    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc5pw\" (UniqueName: \"kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw\") pod \"9ab4676e-738e-4812-bc07-3a5a14eab39e\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") "
Jul 24 13:59:52.798003 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798007    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq56l\" (UniqueName: \"kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l\") pod \"600156c9-59d2-4948-8bf2-5b0b045732cc\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") "
Jul 24 13:59:52.798263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798023    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca\") pod \"9ab4676e-738e-4812-bc07-3a5a14eab39e\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") "
Jul 24 13:59:52.798263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798046    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume\") pod \"600156c9-59d2-4948-8bf2-5b0b045732cc\" (UID: \"600156c9-59d2-4948-8bf2-5b0b045732cc\") "
Jul 24 13:59:52.798263 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798092    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") pod \"9ab4676e-738e-4812-bc07-3a5a14eab39e\" (UID: \"9ab4676e-738e-4812-bc07-3a5a14eab39e\") "
Jul 24 13:59:52.798427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798375    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ab4676e-738e-4812-bc07-3a5a14eab39e" (UID: "9ab4676e-738e-4812-bc07-3a5a14eab39e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:59:52.798475 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.798442    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "600156c9-59d2-4948-8bf2-5b0b045732cc" (UID: "600156c9-59d2-4948-8bf2-5b0b045732cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:59:52.800379 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.800353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ab4676e-738e-4812-bc07-3a5a14eab39e" (UID: "9ab4676e-738e-4812-bc07-3a5a14eab39e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:59:52.800503 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.800482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l" (OuterVolumeSpecName: "kube-api-access-vq56l") pod "600156c9-59d2-4948-8bf2-5b0b045732cc" (UID: "600156c9-59d2-4948-8bf2-5b0b045732cc"). InnerVolumeSpecName "kube-api-access-vq56l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:59:52.800561 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.800517    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw" (OuterVolumeSpecName: "kube-api-access-tc5pw") pod "9ab4676e-738e-4812-bc07-3a5a14eab39e" (UID: "9ab4676e-738e-4812-bc07-3a5a14eab39e"). InnerVolumeSpecName "kube-api-access-tc5pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:59:52.875566 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.875533    2556 generic.go:356] "Generic (PLEG): container finished" podID="9ab4676e-738e-4812-bc07-3a5a14eab39e" containerID="c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac" exitCode=0
Jul 24 13:59:52.875739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.875579    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"
Jul 24 13:59:52.875739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.875614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" event={"ID":"9ab4676e-738e-4812-bc07-3a5a14eab39e","Type":"ContainerDied","Data":"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"}
Jul 24 13:59:52.875739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.875666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj" event={"ID":"9ab4676e-738e-4812-bc07-3a5a14eab39e","Type":"ContainerDied","Data":"f9280a51241b5439d0d690fd59f93c9a8746638fba992c0e51aae5fd0ce37702"}
Jul 24 13:59:52.875739 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.875692    2556 scope.go:117] "RemoveContainer" containerID="c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"
Jul 24 13:59:52.877006 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.876980    2556 generic.go:356] "Generic (PLEG): container finished" podID="600156c9-59d2-4948-8bf2-5b0b045732cc" containerID="8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10" exitCode=0
Jul 24 13:59:52.877096 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.877015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" event={"ID":"600156c9-59d2-4948-8bf2-5b0b045732cc","Type":"ContainerDied","Data":"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"}
Jul 24 13:59:52.877096 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.877043    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"
Jul 24 13:59:52.877096 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.877058    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm" event={"ID":"600156c9-59d2-4948-8bf2-5b0b045732cc","Type":"ContainerDied","Data":"dcd8aa1b57fb047485c2d750212a968e3fd7e487244638c19677c59b1e9c85b6"}
Jul 24 13:59:52.884427 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.884407    2556 scope.go:117] "RemoveContainer" containerID="c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"
Jul 24 13:59:52.884706 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:59:52.884681    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac\": container with ID starting with c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac not found: ID does not exist" containerID="c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"
Jul 24 13:59:52.884804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.884710    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac"} err="failed to get container status \"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac\": rpc error: code = NotFound desc = could not find container \"c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac\": container with ID starting with c60f61add40d02f003d7f2211a21c6d1276e755ee471ebae22fa6230945535ac not found: ID does not exist"
Jul 24 13:59:52.884804 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.884726    2556 scope.go:117] "RemoveContainer" containerID="8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"
Jul 24 13:59:52.893342 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.893326    2556 scope.go:117] "RemoveContainer" containerID="8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"
Jul 24 13:59:52.893590 ip-10-0-136-14 kubenswrapper[2556]: E0724 13:59:52.893573    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10\": container with ID starting with 8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10 not found: ID does not exist" containerID="8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"
Jul 24 13:59:52.893637 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.893597    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10"} err="failed to get container status \"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10\": rpc error: code = NotFound desc = could not find container \"8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10\": container with ID starting with 8253654a3fad5cc28227dba2242c6e09dccd54415898cf1af22b5538e6c01c10 not found: ID does not exist"
Jul 24 13:59:52.899063 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.899045    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ab4676e-738e-4812-bc07-3a5a14eab39e-maas-api-tls\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:52.899123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.899065    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5pw\" (UniqueName: \"kubernetes.io/projected/9ab4676e-738e-4812-bc07-3a5a14eab39e-kube-api-access-tc5pw\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:52.899123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.899074    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq56l\" (UniqueName: \"kubernetes.io/projected/600156c9-59d2-4948-8bf2-5b0b045732cc-kube-api-access-vq56l\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:52.899123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.899083    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab4676e-738e-4812-bc07-3a5a14eab39e-openshift-service-ca\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:52.899123 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.899093    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600156c9-59d2-4948-8bf2-5b0b045732cc-plugins-config-volume\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 13:59:52.902259 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.902235    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:59:52.904422 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.904403    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0a0df7-6c7f556fb6-6hzsj"]
Jul 24 13:59:52.914471 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.914452    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:59:52.918372 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:52.918352    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-0a0df7-5b9c9f5986-w7cwm"]
Jul 24 13:59:53.340681 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:53.340652    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600156c9-59d2-4948-8bf2-5b0b045732cc" path="/var/lib/kubelet/pods/600156c9-59d2-4948-8bf2-5b0b045732cc/volumes"
Jul 24 13:59:53.341072 ip-10-0-136-14 kubenswrapper[2556]: I0724 13:59:53.341059    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab4676e-738e-4812-bc07-3a5a14eab39e" path="/var/lib/kubelet/pods/9ab4676e-738e-4812-bc07-3a5a14eab39e/volumes"
Jul 24 14:00:00.134582 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134546    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"]
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134874    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1c0736-dd1a-4657-901c-5f87be30f364" containerName="revoke-keys"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134885    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1c0736-dd1a-4657-901c-5f87be30f364" containerName="revoke-keys"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134901    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab4676e-738e-4812-bc07-3a5a14eab39e" containerName="maas-api"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134906    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab4676e-738e-4812-bc07-3a5a14eab39e" containerName="maas-api"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134914    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e9208d-5d19-4add-bebc-deec0dd8e161" containerName="maas-api"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134920    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e9208d-5d19-4add-bebc-deec0dd8e161" containerName="maas-api"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134954    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600156c9-59d2-4948-8bf2-5b0b045732cc" containerName="payload-processing"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.134960    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="600156c9-59d2-4948-8bf2-5b0b045732cc" containerName="payload-processing"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.135015    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="600156c9-59d2-4948-8bf2-5b0b045732cc" containerName="payload-processing"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.135022    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab4676e-738e-4812-bc07-3a5a14eab39e" containerName="maas-api"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.135029    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1c0736-dd1a-4657-901c-5f87be30f364" containerName="revoke-keys"
Jul 24 14:00:00.135067 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.135035    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e9208d-5d19-4add-bebc-deec0dd8e161" containerName="maas-api"
Jul 24 14:00:00.139414 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.139398    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"
Jul 24 14:00:00.141902 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.141875    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j482l\""
Jul 24 14:00:00.144415 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.144392    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"]
Jul 24 14:00:00.151396 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.151378    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"
Jul 24 14:00:00.306990 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.306906    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"]
Jul 24 14:00:00.309584 ip-10-0-136-14 kubenswrapper[2556]: W0724 14:00:00.309557    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b705cca_6971_46ef_bffb_31497ec88901.slice/crio-b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b WatchSource:0}: Error finding container b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b: Status 404 returned error can't find the container with id b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b
Jul 24 14:00:00.923860 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.923761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq" event={"ID":"3b705cca-6971-46ef-bffb-31497ec88901","Type":"ContainerStarted","Data":"5a1bc32912e7971bf4ac254984c35766c0b5a68decbc7aa53ab3b0e3b56aac23"}
Jul 24 14:00:00.924068 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.923870    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq" event={"ID":"3b705cca-6971-46ef-bffb-31497ec88901","Type":"ContainerStarted","Data":"b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b"}
Jul 24 14:00:00.939099 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:00.939047    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq" podStartSLOduration=0.939032921 podStartE2EDuration="939.032921ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:00:00.937338534 +0000 UTC m=+3020.161821862" watchObservedRunningTime="2026-07-24 14:00:00.939032921 +0000 UTC m=+3020.163516248"
Jul 24 14:00:01.930337 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:01.930301    2556 generic.go:356] "Generic (PLEG): container finished" podID="3b705cca-6971-46ef-bffb-31497ec88901" containerID="5a1bc32912e7971bf4ac254984c35766c0b5a68decbc7aa53ab3b0e3b56aac23" exitCode=0
Jul 24 14:00:01.930742 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:01.930373    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq" event={"ID":"3b705cca-6971-46ef-bffb-31497ec88901","Type":"ContainerDied","Data":"5a1bc32912e7971bf4ac254984c35766c0b5a68decbc7aa53ab3b0e3b56aac23"}
Jul 24 14:00:03.068609 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:03.068573    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"
Jul 24 14:00:03.942038 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:03.942003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq" event={"ID":"3b705cca-6971-46ef-bffb-31497ec88901","Type":"ContainerDied","Data":"b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b"}
Jul 24 14:00:03.942218 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:03.942049    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6eb34be2bc44f1dab9f6405c61631cf2beeeecb430bf03632c815fb9a95537b"
Jul 24 14:00:03.942218 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:03.942011    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-zdffq"
Jul 24 14:00:04.160368 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:04.160339    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856b56fb59-s9wjl_b15b813a-57cb-47b1-8b35-089b69569975/authorino/0.log"
Jul 24 14:00:11.368405 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:11.368377    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfd75c57-cmwh7_b007d022-84a4-4daa-94e5-d27bf8496da2/copy-manifests/0.log"
Jul 24 14:00:11.374905 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:11.374879    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfd75c57-cmwh7_b007d022-84a4-4daa-94e5-d27bf8496da2/manager/0.log"
Jul 24 14:00:12.155174 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:12.155147    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748345-gmdw6_dc0101c1-3494-4157-9b9b-ad1d6d16650d/cleanup/0.log"
Jul 24 14:00:12.280195 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:12.280168    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748360-zdffq_3b705cca-6971-46ef-bffb-31497ec88901/cleanup/0.log"
Jul 24 14:00:12.401524 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:12.401495    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8pg9x_a1ca0f3c-581f-4fec-ac8f-b4a70ccc291a/postgres/0.log"
Jul 24 14:00:13.809014 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:13.808981    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856b56fb59-s9wjl_b15b813a-57cb-47b1-8b35-089b69569975/authorino/0.log"
Jul 24 14:00:14.172263 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:14.172189    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xsqc2_7ccb5e85-d10e-4dd5-a161-9cedccbf0aed/kuadrant-console-plugin/0.log"
Jul 24 14:00:15.445502 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:15.445472    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76dcd6cb6b-xzl65_2018df84-5adc-4461-9096-f6f6841e6f19/kube-auth-proxy/0.log"
Jul 24 14:00:16.138080 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.138036    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:16.138385 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.138372    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b705cca-6971-46ef-bffb-31497ec88901" containerName="cleanup"
Jul 24 14:00:16.138431 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.138388    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b705cca-6971-46ef-bffb-31497ec88901" containerName="cleanup"
Jul 24 14:00:16.138466 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.138456    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b705cca-6971-46ef-bffb-31497ec88901" containerName="cleanup"
Jul 24 14:00:16.141463 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.141447    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:16.143582 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.143560    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xjspb\""
Jul 24 14:00:16.148635 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.148615    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:16.307915 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.307873    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbf8t\" (UniqueName: \"kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t\") pod \"kuadrant-operator-catalog-4whh5\" (UID: \"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365\") " pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:16.343387 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.343361    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-sz7gg_d02d5035-d048-45de-8ea3-89bb0efd78dc/main/0.log"
Jul 24 14:00:16.408793 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.408712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbf8t\" (UniqueName: \"kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t\") pod \"kuadrant-operator-catalog-4whh5\" (UID: \"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365\") " pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:16.417000 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.416976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbf8t\" (UniqueName: \"kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t\") pod \"kuadrant-operator-catalog-4whh5\" (UID: \"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365\") " pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:16.451026 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.451000    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:16.463282 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.463255    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wnhhc_4ebfaf69-01f6-4719-87dd-1c99b76bdebc/main/0.log"
Jul 24 14:00:16.586549 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.586489    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:16.588901 ip-10-0-136-14 kubenswrapper[2556]: W0724 14:00:16.588865    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7a87d1_0b01_4c1f_b1c1_b80caef1d365.slice/crio-87c96a6decff06552f02e3e508b7e1aa001c35d9f4b60c52ee27c3abb41fa9e7 WatchSource:0}: Error finding container 87c96a6decff06552f02e3e508b7e1aa001c35d9f4b60c52ee27c3abb41fa9e7: Status 404 returned error can't find the container with id 87c96a6decff06552f02e3e508b7e1aa001c35d9f4b60c52ee27c3abb41fa9e7
Jul 24 14:00:16.590321 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.590305    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:00:16.941080 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:16.941046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xlklr_516107c4-ee0d-48d1-aea9-31e33080dd58/main/0.log"
Jul 24 14:00:17.012701 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:17.012673    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" event={"ID":"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365","Type":"ContainerStarted","Data":"87c96a6decff06552f02e3e508b7e1aa001c35d9f4b60c52ee27c3abb41fa9e7"}
Jul 24 14:00:19.024611 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:19.024572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" event={"ID":"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365","Type":"ContainerStarted","Data":"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"}
Jul 24 14:00:26.085106 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.085077    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5mp7r_2ec6639f-9950-4bd0-85d5-a3542940b6aa/global-pull-secret-syncer/0.log"
Jul 24 14:00:26.248769 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.248739    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k4z72_f0445648-3793-4fc2-80b6-1f0b2733536c/konnectivity-agent/0.log"
Jul 24 14:00:26.307287 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.307263    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-14.ec2.internal_fd517d74ad9c69ffc34f61061c12dc4c/haproxy/0.log"
Jul 24 14:00:26.452158 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.452048    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:26.452158 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.452154    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:26.474585 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.474559    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:26.491013 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:26.490970    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" podStartSLOduration=8.205655375 podStartE2EDuration="10.490954725s" totalImagesPullingTime="2.28529935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:00:19.041616932 +0000 UTC m=+3038.266100262" watchObservedRunningTime="2026-07-24 14:00:26.490954725 +0000 UTC m=+3045.715438051"
Jul 24 14:00:27.087283 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:27.087257    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:27.118157 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:27.118124    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:29.075145 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:29.075109    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" podUID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" containerName="registry-server" containerID="cri-o://97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92" gracePeriod=2
Jul 24 14:00:29.326612 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:29.326553    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:29.426718 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:29.426685    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbf8t\" (UniqueName: \"kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t\") pod \"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365\" (UID: \"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365\") "
Jul 24 14:00:29.428921 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:29.428896    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t" (OuterVolumeSpecName: "kube-api-access-dbf8t") pod "8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" (UID: "8d7a87d1-0b01-4c1f-b1c1-b80caef1d365"). InnerVolumeSpecName "kube-api-access-dbf8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:00:29.527366 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:29.527332    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbf8t\" (UniqueName: \"kubernetes.io/projected/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365-kube-api-access-dbf8t\") on node \"ip-10-0-136-14.ec2.internal\" DevicePath \"\""
Jul 24 14:00:30.081458 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.081426    2556 generic.go:356] "Generic (PLEG): container finished" podID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" containerID="97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92" exitCode=0
Jul 24 14:00:30.081850 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.081470    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4whh5"
Jul 24 14:00:30.081850 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.081495    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" event={"ID":"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365","Type":"ContainerDied","Data":"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"}
Jul 24 14:00:30.081850 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.081534    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4whh5" event={"ID":"8d7a87d1-0b01-4c1f-b1c1-b80caef1d365","Type":"ContainerDied","Data":"87c96a6decff06552f02e3e508b7e1aa001c35d9f4b60c52ee27c3abb41fa9e7"}
Jul 24 14:00:30.081850 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.081552    2556 scope.go:117] "RemoveContainer" containerID="97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"
Jul 24 14:00:30.091115 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.091094    2556 scope.go:117] "RemoveContainer" containerID="97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"
Jul 24 14:00:30.091385 ip-10-0-136-14 kubenswrapper[2556]: E0724 14:00:30.091362    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92\": container with ID starting with 97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92 not found: ID does not exist" containerID="97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"
Jul 24 14:00:30.091442 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.091395    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92"} err="failed to get container status \"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92\": rpc error: code = NotFound desc = could not find container \"97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92\": container with ID starting with 97a748e0bfe94284a3f6057fac4faae38a8ed7fef58e579b242351dd0a657a92 not found: ID does not exist"
Jul 24 14:00:30.101859 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.101836    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:30.105947 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:30.105905    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4whh5"]
Jul 24 14:00:31.341846 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:31.341811    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" path="/var/lib/kubelet/pods/8d7a87d1-0b01-4c1f-b1c1-b80caef1d365/volumes"
Jul 24 14:00:31.675421 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:31.675325    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856b56fb59-s9wjl_b15b813a-57cb-47b1-8b35-089b69569975/authorino/0.log"
Jul 24 14:00:31.765614 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:31.765583    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xsqc2_7ccb5e85-d10e-4dd5-a161-9cedccbf0aed/kuadrant-console-plugin/0.log"
Jul 24 14:00:34.280722 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.280694    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cdf6bbc7c-6qfrt_8ccc7612-b377-4303-903f-93b61d3d3725/metrics-server/0.log"
Jul 24 14:00:34.304070 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.304050    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-dtkv6_731e5b78-6d8a-4388-9319-ccee65fc8bfc/monitoring-plugin/0.log"
Jul 24 14:00:34.332096 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.332044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sq6g_35185b15-9219-46c2-88c3-cb3c2ad91761/node-exporter/0.log"
Jul 24 14:00:34.352095 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.352079    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sq6g_35185b15-9219-46c2-88c3-cb3c2ad91761/kube-rbac-proxy/0.log"
Jul 24 14:00:34.372174 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.372153    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sq6g_35185b15-9219-46c2-88c3-cb3c2ad91761/init-textfile/0.log"
Jul 24 14:00:34.640256 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.640185    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/prometheus/0.log"
Jul 24 14:00:34.664497 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.664466    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/config-reloader/0.log"
Jul 24 14:00:34.687524 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.687482    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/thanos-sidecar/0.log"
Jul 24 14:00:34.716738 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.716714    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/kube-rbac-proxy-web/0.log"
Jul 24 14:00:34.739739 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.739719    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/kube-rbac-proxy/0.log"
Jul 24 14:00:34.761680 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.761658    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/kube-rbac-proxy-thanos/0.log"
Jul 24 14:00:34.783403 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.783383    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_559a7ec5-0658-424b-978b-60d599c48a6f/init-config-reloader/0.log"
Jul 24 14:00:34.810036 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.810015    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hm4mf_eb2564cb-7867-49fb-ac14-2f5a3aec6cff/prometheus-operator/0.log"
Jul 24 14:00:34.829297 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.829278    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hm4mf_eb2564cb-7867-49fb-ac14-2f5a3aec6cff/kube-rbac-proxy/0.log"
Jul 24 14:00:34.854289 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.854265    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-866xv_7d6b3bd8-de21-4dd1-80d6-b8acc2a6ddce/prometheus-operator-admission-webhook/0.log"
Jul 24 14:00:34.882905 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.882888    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dff785474-x9xnb_680d50cc-d6f5-43de-955e-c286d1086113/telemeter-client/0.log"
Jul 24 14:00:34.909322 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.909268    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dff785474-x9xnb_680d50cc-d6f5-43de-955e-c286d1086113/reload/0.log"
Jul 24 14:00:34.934214 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.934197    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dff785474-x9xnb_680d50cc-d6f5-43de-955e-c286d1086113/kube-rbac-proxy/0.log"
Jul 24 14:00:34.961213 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.961192    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/thanos-query/0.log"
Jul 24 14:00:34.981337 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:34.981318    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/kube-rbac-proxy-web/0.log"
Jul 24 14:00:35.012440 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.012412    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/kube-rbac-proxy/0.log"
Jul 24 14:00:35.034727 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.034700    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/prom-label-proxy/0.log"
Jul 24 14:00:35.059258 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.059239    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/kube-rbac-proxy-rules/0.log"
Jul 24 14:00:35.081601 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.081577    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd9ff656c-f9tpf_3783e967-19bc-4305-857b-776834ea1d54/kube-rbac-proxy-metrics/0.log"
Jul 24 14:00:35.501376 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.501337    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"]
Jul 24 14:00:35.501762 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.501679    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" containerName="registry-server"
Jul 24 14:00:35.501762 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.501689    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" containerName="registry-server"
Jul 24 14:00:35.501762 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.501745    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7a87d1-0b01-4c1f-b1c1-b80caef1d365" containerName="registry-server"
Jul 24 14:00:35.506150 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.506127    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.508551 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.508522    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgcsj\"/\"openshift-service-ca.crt\""
Jul 24 14:00:35.508669 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.508567    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mgcsj\"/\"default-dockercfg-twfrm\""
Jul 24 14:00:35.509132 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.509113    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgcsj\"/\"kube-root-ca.crt\""
Jul 24 14:00:35.514127 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.514106    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"]
Jul 24 14:00:35.579203 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.579169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-podres\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.579369 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.579223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-proc\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.579369 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.579285    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-lib-modules\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.579369 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.579323    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-sys\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.579502 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.579383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crngw\" (UniqueName: \"kubernetes.io/projected/58b684b3-9cd2-4c82-b35d-ccc821e96b69-kube-api-access-crngw\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680677 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-podres\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-proc\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680706    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-lib-modules\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-sys\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-proc\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crngw\" (UniqueName: \"kubernetes.io/projected/58b684b3-9cd2-4c82-b35d-ccc821e96b69-kube-api-access-crngw\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.680847 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-podres\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.681116 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-lib-modules\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.681116 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.680951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58b684b3-9cd2-4c82-b35d-ccc821e96b69-sys\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.688040 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.688013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crngw\" (UniqueName: \"kubernetes.io/projected/58b684b3-9cd2-4c82-b35d-ccc821e96b69-kube-api-access-crngw\") pod \"perf-node-gather-daemonset-sjwd2\" (UID: \"58b684b3-9cd2-4c82-b35d-ccc821e96b69\") " pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.817665 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.817631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:35.955194 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:35.955162    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"]
Jul 24 14:00:35.958742 ip-10-0-136-14 kubenswrapper[2556]: W0724 14:00:35.958705    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58b684b3_9cd2_4c82_b35d_ccc821e96b69.slice/crio-4505df04cecd3192014c64531b8cea6cac7cfe6e41edd4af491e443a25c47143 WatchSource:0}: Error finding container 4505df04cecd3192014c64531b8cea6cac7cfe6e41edd4af491e443a25c47143: Status 404 returned error can't find the container with id 4505df04cecd3192014c64531b8cea6cac7cfe6e41edd4af491e443a25c47143
Jul 24 14:00:36.111228 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:36.111135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2" event={"ID":"58b684b3-9cd2-4c82-b35d-ccc821e96b69","Type":"ContainerStarted","Data":"d3941fa81d9276af400010f8e918de4ea973af7c8b6effc404e169d26cb21e6e"}
Jul 24 14:00:36.111228 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:36.111193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2" event={"ID":"58b684b3-9cd2-4c82-b35d-ccc821e96b69","Type":"ContainerStarted","Data":"4505df04cecd3192014c64531b8cea6cac7cfe6e41edd4af491e443a25c47143"}
Jul 24 14:00:36.111422 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:36.111265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:36.127066 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:36.127017    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2" podStartSLOduration=1.127002149 podStartE2EDuration="1.127002149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:00:36.126297054 +0000 UTC m=+3055.350780381" watchObservedRunningTime="2026-07-24 14:00:36.127002149 +0000 UTC m=+3055.351485476"
Jul 24 14:00:38.958484 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:38.958451    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkbk9_be8f59de-48d2-49f1-b36e-846c6ccfab47/dns/0.log"
Jul 24 14:00:38.979376 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:38.979357    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkbk9_be8f59de-48d2-49f1-b36e-846c6ccfab47/kube-rbac-proxy/0.log"
Jul 24 14:00:39.130445 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:39.130416    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lwjjd_672e4e80-f74f-4e3c-a20f-3167fe4970b5/dns-node-resolver/0.log"
Jul 24 14:00:39.740327 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:39.740299    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l4qff_0053d784-2bcf-4674-a6fa-2301aa65cab3/node-ca/0.log"
Jul 24 14:00:40.915760 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:40.915731    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76dcd6cb6b-xzl65_2018df84-5adc-4461-9096-f6f6841e6f19/kube-auth-proxy/0.log"
Jul 24 14:00:41.909989 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:41.909924    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvjrb_1c6ed8d2-0deb-404f-ab68-6972a2ec9d03/serve-healthcheck-canary/0.log"
Jul 24 14:00:42.127438 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:42.127413    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mgcsj/perf-node-gather-daemonset-sjwd2"
Jul 24 14:00:42.589720 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:42.589687    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrjqx_f46777f9-0473-4e96-9624-bdd5fb137698/kube-rbac-proxy/0.log"
Jul 24 14:00:42.610803 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:42.610776    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrjqx_f46777f9-0473-4e96-9624-bdd5fb137698/exporter/0.log"
Jul 24 14:00:42.634724 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:42.634702    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vrjqx_f46777f9-0473-4e96-9624-bdd5fb137698/extractor/0.log"
Jul 24 14:00:45.916309 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:45.916274    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfd75c57-cmwh7_b007d022-84a4-4daa-94e5-d27bf8496da2/manager/0.log"
Jul 24 14:00:45.940613 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:45.940586    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfd75c57-cmwh7_b007d022-84a4-4daa-94e5-d27bf8496da2/copy-manifests/0.log"
Jul 24 14:00:46.233231 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:46.233146    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76dcd6cb6b-xzl65_2018df84-5adc-4461-9096-f6f6841e6f19/kube-auth-proxy/0.log"
Jul 24 14:00:47.385492 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:47.385462    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d6cbddddd-7nz8b_8bafc74a-e0ef-4a25-bca8-cda204c4f50d/manager/0.log"
Jul 24 14:00:47.448420 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:47.448394    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zwtlp_e838ad2c-fb30-4bca-850f-4d34636c4d31/openshift-lws-operator/0.log"
Jul 24 14:00:48.015558 ip-10-0-136-14 kubenswrapper[2556]: I0724 14:00:48.015528    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76dcd6cb6b-xzl65_2018df84-5adc-4461-9096-f6f6841e6f19/kube-auth-proxy/0.log"