Sep 11 21:17:30.623389 ip-10-0-135-133 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:17:31.041364 ip-10-0-135-133 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:17:31.041364 ip-10-0-135-133 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:17:31.041364 ip-10-0-135-133 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:17:31.041364 ip-10-0-135-133 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:17:31.041364 ip-10-0-135-133 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:17:31.044293 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.044202    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:17:31.047462 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047447    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.047462 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047462    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047466    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047470    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047473    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047476    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047479    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047482    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047484    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047494    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047497    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047500    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047502    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047506    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047508    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047511    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047513    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047516    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047518    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047521    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047523    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.047525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047526    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047529    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047532    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047535    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047538    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047540    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047543    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047545    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047550    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047554    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047557    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047560    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047563    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047566    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047569    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047572    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047574    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047577    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047580    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.048096 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047582    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047585    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047587    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047589    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047592    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047594    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047604    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047608    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047612    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047614    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047617    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047620    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047622    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047625    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047627    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047631    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047636    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047640    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047643    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.048578 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047646    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047649    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047652    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047655    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047658    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047661    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047663    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047666    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047668    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047671    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047674    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047676    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047679    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047682    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047685    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047687    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047690    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047692    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047695    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.049038 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047697    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047700    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047702    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047705    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047708    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047712    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047715    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047717    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047720    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047723    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047726    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047729    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047732    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047734    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047737    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047739    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047742    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047744    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047747    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047749    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.049555 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.047752    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048441    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048447    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048451    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048454    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048457    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048460    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048463    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048465    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048468    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048471    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048473    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048476    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048478    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048481    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048483    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048486    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048488    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048491    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048493    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.050022 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048496    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048500    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048504    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048508    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048511    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048514    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048516    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048519    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048521    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048523    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048526    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048528    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048531    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048533    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048536    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048539    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048541    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048544    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048547    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.050538 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048549    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048552    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048555    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048557    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048560    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048563    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048565    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048568    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048570    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048574    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048578    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048581    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048584    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048587    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048591    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048593    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048596    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048599    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048602    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.050993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048604    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048607    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048609    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048612    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048615    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048619    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048622    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048625    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048628    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048631    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048633    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048636    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048639    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048641    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048646    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048648    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048651    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048653    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048656    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.051479 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048658    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048660    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048663    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048665    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048668    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048671    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048673    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048675    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048678    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048681    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048684    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048686    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048689    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048691    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048694    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048696    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048699    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048701    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048703    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048706    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.051959 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048708    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048711    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.048713    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049051    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049055    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049058    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049060    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049064    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049066    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049069    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049071    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049074    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049076    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049080    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049083    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049086    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049089    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049092    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049094    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.052488 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049097    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049099    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049102    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049105    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049125    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049128    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049131    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049134    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049137    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049140    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049143    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049145    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049148    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049151    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049153    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049156    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049158    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049161    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049164    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049166    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.061012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049171    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049174    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049177    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049180    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049182    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049185    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049187    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049190    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049192    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049195    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049198    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049200    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049203    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049210    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049213    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049216    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049218    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049221    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049223    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049226    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.061577 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049228    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049231    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049233    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049236    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049240    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049243    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049246    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049249    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049251    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049254    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049257    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049260    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049262    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049265    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049268    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049272    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049274    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049277    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049279    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049281    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.062069 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049284    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049286    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049289    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049292    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049294    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049298    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049301    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049303    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049306    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049308    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049311    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049313    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049316    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049318    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049321    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049326    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049329    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049331    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.062628 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049333    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049336    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049339    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.049341    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049415    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049468    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049476    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049480    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049484    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049488    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049492    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049496    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049499    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049502    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049507    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049510    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049513    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049516    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049520    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049523    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049526    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049529    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049532    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:17:31.063076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049535    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049538    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049541    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049544    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049547    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049551    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049554    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049557    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049560    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049563    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049566    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049569    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049572    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049576    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049578    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049581    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049584    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049587    2580 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049590    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049594    2580 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049597    2580 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049600    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049603    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049607    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:17:31.063684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049611    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049614    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049616    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049620    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049623    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049626    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049629    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049631    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049635    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049638    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049640    2580 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049644    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049647    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049650    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049653    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049656    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049659    2580 flags.go:64] FLAG: --help="false"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049662    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049666    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049668    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049671    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049674    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049677    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049680    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:17:31.064383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049683    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049686    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049688    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049691    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049694    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049697    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049700    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049705    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049708    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049711    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049713    2580 flags.go:64] FLAG: --lock-file=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049717    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049719    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049722    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049727    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049730    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049733    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049736    2580 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049738    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049742    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049745    2580 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049747    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049751    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049754    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049758    2580 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:17:31.064981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049761    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049764    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049767    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049770    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049773    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049776    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049778    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049785    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049788    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049792    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049795    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049798    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049803    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049806    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049810    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049814    2580 flags.go:64] FLAG: --port="10250"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049817    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049820    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08db47720e692d7da"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049823    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049826    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049829    2580 flags.go:64] FLAG: --register-node="true"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049832    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049835    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049838    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:17:31.065650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049841    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049843    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049850    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049853    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049856    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049859    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049862    2580 flags.go:64] FLAG: --runonce="false"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049865    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049868    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049871    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049873    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049876    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049879    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049882    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049885    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049888    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049890    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049893    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049896    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049899    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049902    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049905    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049910    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049914    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049963    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:17:31.066410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049968    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049971    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049974    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049978    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049981    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049984    2580 flags.go:64] FLAG: --v="2"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049988    2580 flags.go:64] FLAG: --version="false"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049992    2580 flags.go:64] FLAG: --vmodule=""
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049995    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.049998    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050128    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050132    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050136    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050139    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050142    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050145    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050147    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050150    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050153    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050156    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050159    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.067084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050161    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050164    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050168    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050171    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050173    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050176    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050178    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050181    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050184    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050188    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050190    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050193    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050196    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050199    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050203    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050206    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050209    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050212    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050215    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.067634 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050217    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050221    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050223    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050226    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050229    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050231    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050234    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050237    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050241    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050244    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050247    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050250    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050253    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050255    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050258    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050262    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050264    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050267    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050269    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.068418 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050272    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050275    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050278    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050282    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050285    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050287    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050290    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050292    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050295    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050298    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050300    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050303    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050305    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050308    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050310    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050313    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050316    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050318    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050321    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050323    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.069004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050326    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050329    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050331    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050334    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050336    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050339    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050341    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050344    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050349    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050351    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050354    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050357    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050359    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050362    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050364    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050369    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050372    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050374    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050377    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050380    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.069518 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050382    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050385    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050387    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050390    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050394    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050397    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050400    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050403    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050406    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.050409    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.050414    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058183    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058198    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058217    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:17:31.069999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058225    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058280    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058285    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058288    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058291    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058296    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058300    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058304    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058307    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058310    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058312    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058315    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058318    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058321    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058326    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058329    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058331    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058334    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.070496 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058336    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058339    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058343    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058346    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058350    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058353    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058355    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058358    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058360    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058363    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058365    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058369    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058372    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058375    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058378    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058380    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058383    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058385    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058388    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058390    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.070964 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058393    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058396    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058399    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058401    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058404    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058406    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058409    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058411    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058414    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058416    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058419    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058421    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058424    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058427    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058430    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058432    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058435    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058437    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058439    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058442    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.071535 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058445    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058447    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058449    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058453    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058456    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058459    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058461    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058464    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058466    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058469    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058471    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058474    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058476    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058479    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058481    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058484    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058486    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058490    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058494    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.072055 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058497    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058500    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058502    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058505    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058507    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058510    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058512    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058515    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058517    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058519    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058522    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058525    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058528    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058530    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058533    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058536    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058539    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058542    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058545    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058547    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.072543 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058550    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058552    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058557    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058668    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058672    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058675    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058678    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058681    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058684    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058687    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058690    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058694    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058697    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058700    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058703    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:17:31.073229 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058705    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058708    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058713    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058716    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058718    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058721    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058723    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058726    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058729    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058732    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058735    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058738    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058741    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058743    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058747    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058751    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058754    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058757    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058761    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:17:31.073720 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058764    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058766    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058769    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058771    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058774    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058776    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058779    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058781    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058784    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058786    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058789    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058791    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058794    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058796    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058799    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058802    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058804    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058806    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058809    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058811    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:17:31.074368 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058814    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058816    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058819    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058822    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058824    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058828    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058830    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058833    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058835    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058838    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058842    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058845    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058848    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058851    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058853    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058856    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058858    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058860    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058863    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:17:31.074874 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058865    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058868    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058870    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058873    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058875    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058878    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058880    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058883    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058885    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058888    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058890    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058892    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058895    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058897    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058900    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058902    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058905    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058907    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058910    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058913    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:17:31.075445 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058915    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058917    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058920    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058923    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058925    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058928    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058930    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:31.058933    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.058938    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.059036    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.062921    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.064032    2580 server.go:1019] "Starting client certificate rotation"
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.064158    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:17:31.075993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.064211    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:17:31.076391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.066835    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:17:31.076391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.070253    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:17:31.080262 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.080246    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:17:31.085434 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.085420    2580 log.go:25] "Validated CRI v1 image API"
Sep 11 21:17:31.086176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.086162    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:17:31.089443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.089425    2580 fs.go:135] Filesystem UUIDs: map[391541d7-2a1c-49bf-aa6f-9d6e05c9bd54:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8a90ff8a-44d6-46ac-9792-07d26f6becff:/dev/nvme0n1p3]
Sep 11 21:17:31.089503 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.089442    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:17:31.095496 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.095387    2580 manager.go:216] Machine: {Timestamp:2026-09-11 21:17:31.093942706 +0000 UTC m=+0.363664894 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100948 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2701ee1eb31da05355f412e668c569 SystemUUID:ec2701ee-1eb3-1da0-5355-f412e668c569 BootID:f57942d6-62f0-4a87-943b-770e33f43137 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:18:50:94:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:18:50:94:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e2:9e:a2:e3:6c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:17:31.095496 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.095492    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:17:31.095624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.095608    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:17:31.099148 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.099129    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:17:31.099274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.099151    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:17:31.100045 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.100035    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:17:31.100076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.100047    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:17:31.100076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.100054    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:17:31.100818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.100808    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:17:31.101990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.101979    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:17:31.104510 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.104492    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:17:31.106199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106188    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:17:31.106314 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106304    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:17:31.106342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106334    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:17:31.106375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106349    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:17:31.106406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106374    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:17:31.106406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106397    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:17:31.106470 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106415    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:17:31.106831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106807    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:17:31.106831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.106828    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:17:31.109335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.109302    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:17:31.113450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.113433    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:17:31.113527 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.113473    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:17:31.115567 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115553    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:17:31.115567 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115571    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:17:31.115567 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115577    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115582    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115590    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115595    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115619    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115624    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115631    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115637    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115645    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:17:31.115709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.115650    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:17:31.116473 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.116463    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:17:31.116473 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.116473    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:17:31.118039 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.118018    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:17:31.118039 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.118019    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:17:31.120330 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.120317    2580 server.go:1268] "Started kubelet"
Sep 11 21:17:31.120425 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.120403    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:17:31.120788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.120740    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:17:31.120837 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.120811    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:17:31.121088 ip-10-0-135-133 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:17:31.122097 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.122081    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 21:17:31.122394 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.122380    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:17:31.123024 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.123011    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:17:31.127736 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.127719    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:17:31.129246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.129226    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:17:31.129897 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.129879    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:17:31.130093 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.130049    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.130783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.130767    2580 factory.go:55] Registering systemd factory
Sep 11 21:17:31.130783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.130785    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:17:31.130915 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.130804    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:17:31.130989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.130804    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:17:31.131039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.130994    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:17:31.131039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131020    2580 factory.go:153] Registering CRI-O factory
Sep 11 21:17:31.131039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131034    2580 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:17:31.131182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131134    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:17:31.131182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131153    2580 factory.go:103] Registering Raw factory
Sep 11 21:17:31.131182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131169    2580 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:17:31.131341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131243    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:17:31.131341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131252    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:17:31.131607 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.131583    2580 manager.go:349] Starting recovery of all containers
Sep 11 21:17:31.132667 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.131470    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-133.ec2.internal.18d460dc43d8d7e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-133.ec2.internal,UID:ip-10-0-135-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-133.ec2.internal,},FirstTimestamp:2026-09-11 21:17:31.120285665 +0000 UTC m=+0.390007851,LastTimestamp:2026-09-11 21:17:31.120285665 +0000 UTC m=+0.390007851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-133.ec2.internal,}"
Sep 11 21:17:31.141825 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.141810    2580 manager.go:354] Recovery completed
Sep 11 21:17:31.142593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.142569    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:17:31.143397 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.143373    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 21:17:31.143629 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.143583    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 21:17:31.148242 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.148226    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.149440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.149423    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f6md4"
Sep 11 21:17:31.150293 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150276    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.150391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150307    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.150391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150321    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.150874 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150856    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:17:31.150874 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150868    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.150874 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150872    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:17:31.150992 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.150888    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:17:31.151485 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.151472    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.151549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.151494    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.151549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.151503    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.152929 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.152918    2580 policy_none.go:49] "None policy: Start"
Sep 11 21:17:31.152965 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.152934    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:17:31.152965 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.152943    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:17:31.154758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.154747    2580 policy_none.go:47] "Start"
Sep 11 21:17:31.159660 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.159580    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-133.ec2.internal.18d460dc45a2b3c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-133.ec2.internal,UID:ip-10-0-135-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-133.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-133.ec2.internal,},FirstTimestamp:2026-09-11 21:17:31.150291908 +0000 UTC m=+0.420014095,LastTimestamp:2026-09-11 21:17:31.150291908 +0000 UTC m=+0.420014095,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-133.ec2.internal,}"
Sep 11 21:17:31.159768 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.159687    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f6md4"
Sep 11 21:17:31.192750 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.192736    2580 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.192775    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.192787    2580 server.go:87] "Starting device plugin registration server"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.192998    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.193009    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.193132    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.193235    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.193244    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.193763    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:17:31.214474 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.193818    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.229348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.229300    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:17:31.229348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.229347    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:17:31.229529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.229368    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:17:31.229529 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.229431    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 21:17:31.231698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.231684    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:31.293880 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.293864    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.294627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.294613    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.294711 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.294643    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.294711 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.294662    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.294711 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.294690    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.304961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.304947    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.304961 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.304965    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-133.ec2.internal\": node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.330079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.330057    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"]
Sep 11 21:17:31.330169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.330143    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.330794 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.330781    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.330863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.330805    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.330863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.330816    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.331916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.331901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.331967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.331925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.332865 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.332854    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.332982 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.332967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.333034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333000    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.333492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333478    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.333562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333500    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.333562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333510    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.333562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333542    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.333562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333563    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.333745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.333577    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.335679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.335662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.335760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.335691    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:31.336383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.336366    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:31.336454 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.336396    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:31.336454 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.336405    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:31.341820 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.341805    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.361646 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.361625    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-133.ec2.internal\" not found" node="ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.365821 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.365807    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-133.ec2.internal\" not found" node="ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.432415 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.432387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.432566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.432418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.432566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.432466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7628f9440b1b266be07ea0dcc5e5880-config\") pod \"kube-apiserver-proxy-ip-10-0-135-133.ec2.internal\" (UID: \"a7628f9440b1b266be07ea0dcc5e5880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.432566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.432507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.432566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.432516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f90cc286af7f28dc7b2553d2cebe6176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal\" (UID: \"f90cc286af7f28dc7b2553d2cebe6176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.442503 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.442482    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.533615 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.533561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7628f9440b1b266be07ea0dcc5e5880-config\") pod \"kube-apiserver-proxy-ip-10-0-135-133.ec2.internal\" (UID: \"a7628f9440b1b266be07ea0dcc5e5880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.533615 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.533613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7628f9440b1b266be07ea0dcc5e5880-config\") pod \"kube-apiserver-proxy-ip-10-0-135-133.ec2.internal\" (UID: \"a7628f9440b1b266be07ea0dcc5e5880\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.543051 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.543031    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.643283 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.643253    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.664412 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.664395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.668427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.668412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:31.744356 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.744200    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.844806 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.844722    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.945210 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:31.945185    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:31.983281 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:31.983265    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:32.045968 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:32.045943    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:32.064104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.064086    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:17:32.064214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.064196    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:17:32.064258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.064243    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:17:32.146202 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:32.146142    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:32.163257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.163226    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:12:32 +0000 UTC" deadline="2028-05-07 18:46:26.749344966 +0000 UTC"
Sep 11 21:17:32.163257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.163252    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14493h28m54.586095328s"
Sep 11 21:17:32.227518 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.227486    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:17:32.228747 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.228726    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:17:32.228870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.228775    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:17:32.228870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.228790    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:17:32.246665 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:32.246646    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:32.336554 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:32.336523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90cc286af7f28dc7b2553d2cebe6176.slice/crio-7b8024df2ac0b9dd651a57af348e2c2ee927a1ae2a25e7435ddbec94a81c350e WatchSource:0}: Error finding container 7b8024df2ac0b9dd651a57af348e2c2ee927a1ae2a25e7435ddbec94a81c350e: Status 404 returned error can't find the container with id 7b8024df2ac0b9dd651a57af348e2c2ee927a1ae2a25e7435ddbec94a81c350e
Sep 11 21:17:32.338229 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.338133    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:32.340417 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.340404    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:17:32.347290 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:32.347272    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-133.ec2.internal\" not found"
Sep 11 21:17:32.402424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.402047    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:32.430402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.430382    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:32.441919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.441900    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:17:32.443662 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.443650    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal"
Sep 11 21:17:32.453826 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:32.453812    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:17:34.202634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.202604    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:34.255852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.255825    2580 apiserver.go:52] "Watching apiserver"
Sep 11 21:17:34.263409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.263386    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:17:34.264668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.264643    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-mc442","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal","openshift-multus/multus-6hbc5","openshift-multus/multus-additional-cni-plugins-gx6bl","openshift-network-operator/iptables-alerter-wdwrk","openshift-ovn-kubernetes/ovnkube-node-fbxcq","kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7sqxn","openshift-dns/node-resolver-6mmbt","openshift-image-registry/node-ca-j4mfs","openshift-multus/network-metrics-daemon-tc88x","openshift-network-diagnostics/network-check-target-96jdf","kube-system/global-pull-secret-syncer-p7n7b"]
Sep 11 21:17:34.267420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.267396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.269705 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.269686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:34.269788 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.269748    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:34.270121 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.270105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p7bdp\""
Sep 11 21:17:34.270890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.270866    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:17:34.270978 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.270910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.271153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.271117    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:17:34.271218 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.271204    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:17:34.271271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.271214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.271449 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.271426    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:17:34.272010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.271991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.272097 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.272050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:34.274624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.274604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.276485 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.276467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.277945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.277866    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:17:34.278833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.278810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.280050 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.279997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67vfc\""
Sep 11 21:17:34.280128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rjkck\""
Sep 11 21:17:34.280128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:17:34.280236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280196    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.280290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.280290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280028    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:17:34.280383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280346    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:17:34.280545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.280491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:17:34.281184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.281161    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:17:34.281342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.281328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.281733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.281715    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.282030 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.282008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wzts6\""
Sep 11 21:17:34.282374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.282358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.284636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.284595    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pgzd8\""
Sep 11 21:17:34.284636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.284624    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.285819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.285147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.286238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.286085    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.287843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.287823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.288648 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.288632    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:17:34.288737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.288653    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:17:34.289516 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.289479    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:17:34.290252 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-58xs2\""
Sep 11 21:17:34.290585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-env-overrides\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.290685 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-conf-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.290685 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-multus-daemon-config\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.290685 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.290832 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r4szf\""
Sep 11 21:17:34.290832 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.290832 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-slash\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.290967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-systemd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.290967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-config\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.290967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovn-node-metrics-cert\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.291120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.290980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkqv\" (UniqueName: \"kubernetes.io/projected/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-kube-api-access-wxkqv\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.291120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-cnibin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.291219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-hostroot\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.291219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-cnibin\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.291307 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.291307 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-log-socket\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.291400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.291400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-script-lib\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.291492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-kubelet-config\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.291492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.291668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-cni-binary-copy\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.291737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.291714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-k8s-cni-cncf-io\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.292444 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.292424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-node-log\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.292524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.292461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.293065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.293041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.293141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.293113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:17:34.294570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.293372    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.294570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.294570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.292493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-multus\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.294570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-kubelet\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.294570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6qr\" (UniqueName: \"kubernetes.io/projected/5851258e-a72c-48ff-86db-af65f9b5f70e-kube-api-access-lp6qr\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.294924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9q2\" (UniqueName: \"kubernetes.io/projected/03b3e290-3fd0-4911-9755-76b268887c68-kube-api-access-hq9q2\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.294924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-system-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.294924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.294924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-netns\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.294924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-etc-kubernetes\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.295197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.294951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b3e290-3fd0-4911-9755-76b268887c68-host-slash\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.295197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-systemd-units\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.295197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-netns\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.295197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-system-cni-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.295197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-kubelet\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-socket-dir-parent\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-bin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-multus-certs\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.295960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b3e290-3fd0-4911-9755-76b268887c68-iptables-alerter-script\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296099    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z4rl8\""
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-var-lib-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-etc-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-ovn\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-netd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-dbus\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvttv\" (UniqueName: \"kubernetes.io/projected/a5747e3b-6c03-4944-a051-520a9910f76e-kube-api-access-dvttv\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-bin\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.296822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.296981 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.296943    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:34.297845 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.297825    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:17:34.297931 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.297843    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.299195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.299177    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:17:34.299460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.299438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzt2q\""
Sep 11 21:17:34.327177 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.327158    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-96lk9"
Sep 11 21:17:34.336624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.336594    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-96lk9"
Sep 11 21:17:34.379320 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.379298    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:17:34.381417 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.381376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal" event={"ID":"f90cc286af7f28dc7b2553d2cebe6176","Type":"ContainerStarted","Data":"7b8024df2ac0b9dd651a57af348e2c2ee927a1ae2a25e7435ddbec94a81c350e"}
Sep 11 21:17:34.382490 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.382466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal" event={"ID":"a7628f9440b1b266be07ea0dcc5e5880","Type":"ContainerStarted","Data":"3e8c852128fce496a665e2057ff84d0136230d30574210b38f227bb12d9ee5e0"}
Sep 11 21:17:34.397328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.397410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-systemd\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.397410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-sys\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.397410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-lib-modules\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-system-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b3e290-3fd0-4911-9755-76b268887c68-host-slash\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-system-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-tuned\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-host\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b3e290-3fd0-4911-9755-76b268887c68-host-slash\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-kubernetes\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.397628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-bin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b3e290-3fd0-4911-9755-76b268887c68-iptables-alerter-script\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.397687    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-bin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5mj\" (UniqueName: \"kubernetes.io/projected/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-kube-api-access-cg5mj\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.397760    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:34.897726438 +0000 UTC m=+3.020058373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-tmp-dir\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-etc-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-ovn\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-netd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-etc-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-netd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-ovn\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-dbus\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.398034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkct\" (UniqueName: \"kubernetes.io/projected/fdc27677-ab18-4b21-b0d2-994d107b7a9a-kube-api-access-wfkct\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.397988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-dbus\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovn-node-metrics-cert\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkqv\" (UniqueName: \"kubernetes.io/projected/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-kube-api-access-wxkqv\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-hostroot\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-cnibin\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398214    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-script-lib\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-kubelet-config\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b3e290-3fd0-4911-9755-76b268887c68-iptables-alerter-script\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-cni-binary-copy\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-k8s-cni-cncf-io\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-ovn-kubernetes\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-multus\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-kubelet\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.398568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6qr\" (UniqueName: \"kubernetes.io/projected/5851258e-a72c-48ff-86db-af65f9b5f70e-kube-api-access-lp6qr\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-kube-api-access-7xqg7\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-socket-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-sys-fs\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-netns\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-etc-kubernetes\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398572    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysconfig\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-hostroot\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-systemd-units\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-netns\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-cnibin\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-system-cni-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-modprobe-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-serviceca\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.399210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-kubelet\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-netns\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-run-netns\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-system-cni-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-etc-kubernetes\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-socket-dir-parent\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-multus-certs\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-conf\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgmg\" (UniqueName: \"kubernetes.io/projected/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-kube-api-access-rvgmg\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-socket-dir-parent\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.398873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-kubelet\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-var-lib-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-systemd-units\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvttv\" (UniqueName: \"kubernetes.io/projected/a5747e3b-6c03-4944-a051-520a9910f76e-kube-api-access-dvttv\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-script-lib\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.399863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-multus-certs\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5851258e-a72c-48ff-86db-af65f9b5f70e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-bin\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-env-overrides\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-conf-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75b7b403-2cec-4496-b4d5-11ac365f0305-kubelet-config\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-multus-daemon-config\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-cni-multus\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9q2\" (UniqueName: \"kubernetes.io/projected/03b3e290-3fd0-4911-9755-76b268887c68-kube-api-access-hq9q2\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-run-k8s-cni-cncf-io\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-var-lib-openvswitch\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df250077-d04a-4193-8ab7-758369c6af18-agent-certs\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-device-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.400403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-slash\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-systemd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-config\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-conf-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-env-overrides\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-run-systemd\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-cni-bin\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5851258e-a72c-48ff-86db-af65f9b5f70e-cni-binary-copy\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-cnibin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.399984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-host-slash\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-host-var-lib-kubelet\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-run\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-cnibin\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-var-lib-kubelet\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rht\" (UniqueName: \"kubernetes.io/projected/dc5ab34a-c777-443c-b624-4726afafe05d-kube-api-access-42rht\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-log-socket\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-host\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-tmp\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5747e3b-6c03-4944-a051-520a9910f76e-multus-cni-dir\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df250077-d04a-4193-8ab7-758369c6af18-konnectivity-ca\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-log-socket\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-registration-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovnkube-config\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-hosts-file\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-node-log\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-multus-daemon-config\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-node-log\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.401927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.400614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5747e3b-6c03-4944-a051-520a9910f76e-cni-binary-copy\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.403160 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.403131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-ovn-node-metrics-cert\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.406546 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.406526    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:34.406640 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.406551    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:34.406640 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.406566    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:34.406755 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.406644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:34.906625861 +0000 UTC m=+3.028957783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:34.417000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.408584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkqv\" (UniqueName: \"kubernetes.io/projected/ca456b7e-4441-4ed3-9308-4ec9e34c45aa-kube-api-access-wxkqv\") pod \"ovnkube-node-fbxcq\" (UID: \"ca456b7e-4441-4ed3-9308-4ec9e34c45aa\") " pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.417000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.409478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9q2\" (UniqueName: \"kubernetes.io/projected/03b3e290-3fd0-4911-9755-76b268887c68-kube-api-access-hq9q2\") pod \"iptables-alerter-wdwrk\" (UID: \"03b3e290-3fd0-4911-9755-76b268887c68\") " pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.417000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.409893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvttv\" (UniqueName: \"kubernetes.io/projected/a5747e3b-6c03-4944-a051-520a9910f76e-kube-api-access-dvttv\") pod \"multus-6hbc5\" (UID: \"a5747e3b-6c03-4944-a051-520a9910f76e\") " pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.417000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.410073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6qr\" (UniqueName: \"kubernetes.io/projected/5851258e-a72c-48ff-86db-af65f9b5f70e-kube-api-access-lp6qr\") pod \"multus-additional-cni-plugins-gx6bl\" (UID: \"5851258e-a72c-48ff-86db-af65f9b5f70e\") " pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.500984 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.500914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysconfig\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.500984 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.500956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-modprobe-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.500984 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.500972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-serviceca\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.500996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-conf\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysconfig\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgmg\" (UniqueName: \"kubernetes.io/projected/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-kube-api-access-rvgmg\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-modprobe-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df250077-d04a-4193-8ab7-758369c6af18-agent-certs\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-conf\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-device-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.501227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-run\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-var-lib-kubelet\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.501201    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rht\" (UniqueName: \"kubernetes.io/projected/dc5ab34a-c777-443c-b624-4726afafe05d-kube-api-access-42rht\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.501321    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:17:35.001307352 +0000 UTC m=+3.123639251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-host\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-tmp\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-run\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df250077-d04a-4193-8ab7-758369c6af18-konnectivity-ca\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-var-lib-kubelet\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-registration-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-hosts-file\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-systemd\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-host\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-sys\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.501671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-serviceca\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-lib-modules\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-sys\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-tuned\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-host\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-hosts-file\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-kubernetes\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501687    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-kubernetes\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5mj\" (UniqueName: \"kubernetes.io/projected/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-kube-api-access-cg5mj\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-tmp-dir\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-device-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkct\" (UniqueName: \"kubernetes.io/projected/fdc27677-ab18-4b21-b0d2-994d107b7a9a-kube-api-access-wfkct\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-kube-api-access-7xqg7\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-lib-modules\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-socket-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-host\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.502469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-registration-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-sys-fs\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.501996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-sys-fs\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-systemd\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df250077-d04a-4193-8ab7-758369c6af18-konnectivity-ca\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-tmp-dir\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc5ab34a-c777-443c-b624-4726afafe05d-socket-dir\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.503529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.502470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-sysctl-d\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.503987 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.503964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-etc-tuned\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.504320 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.504299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df250077-d04a-4193-8ab7-758369c6af18-agent-certs\") pod \"konnectivity-agent-mc442\" (UID: \"df250077-d04a-4193-8ab7-758369c6af18\") " pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.504650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.504626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-tmp\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.509345 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.509326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgmg\" (UniqueName: \"kubernetes.io/projected/9bf02f91-8277-4937-a4cf-eda0b2e0c0d8-kube-api-access-rvgmg\") pod \"node-resolver-6mmbt\" (UID: \"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8\") " pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.512673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.512649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5mj\" (UniqueName: \"kubernetes.io/projected/c2a51e6e-8e64-4ae8-a1c4-9207f0322e19-kube-api-access-cg5mj\") pod \"node-ca-j4mfs\" (UID: \"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19\") " pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.513298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.513276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqg7\" (UniqueName: \"kubernetes.io/projected/4b8ec2fb-6130-4f3e-adda-0a8fe35bad02-kube-api-access-7xqg7\") pod \"tuned-7sqxn\" (UID: \"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02\") " pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.513398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.513382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkct\" (UniqueName: \"kubernetes.io/projected/fdc27677-ab18-4b21-b0d2-994d107b7a9a-kube-api-access-wfkct\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:34.513622 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.513591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rht\" (UniqueName: \"kubernetes.io/projected/dc5ab34a-c777-443c-b624-4726afafe05d-kube-api-access-42rht\") pod \"aws-ebs-csi-driver-node-h89jr\" (UID: \"dc5ab34a-c777-443c-b624-4726afafe05d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.580852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.580829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:34.592411 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.592392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hbc5"
Sep 11 21:17:34.599993 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.599974    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gx6bl"
Sep 11 21:17:34.613619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.613596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wdwrk"
Sep 11 21:17:34.621251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.621233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7sqxn"
Sep 11 21:17:34.628850 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.628835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:34.635547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.635531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr"
Sep 11 21:17:34.643064 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.643045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mmbt"
Sep 11 21:17:34.650631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.650613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j4mfs"
Sep 11 21:17:34.903586 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.903495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:34.903733 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.903670    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:34.903777 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:34.903746    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:35.903726117 +0000 UTC m=+4.026058019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:34.955021 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:34.954998    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:17:35.004136 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.004104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:35.004271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.004142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:35.004271 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004247    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:35.004271 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004260    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:35.004437 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004283    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:35.004437 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004293    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:35.004437 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004303    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:17:36.004287002 +0000 UTC m=+4.126618915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:35.004437 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.004347    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:36.004328029 +0000 UTC m=+4.126659928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:35.341021 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.340927    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:12:34 +0000 UTC" deadline="2028-06-26 14:15:08.519600636 +0000 UTC"
Sep 11 21:17:35.341021 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.340966    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15688h57m33.178639039s"
Sep 11 21:17:35.377254 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.377229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:35.377385 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.377335    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:35.466459 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.466399    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca456b7e_4441_4ed3_9308_4ec9e34c45aa.slice/crio-93afde8159b211141399118c63002c7a524844c30a624ae3c363ef912ba46252 WatchSource:0}: Error finding container 93afde8159b211141399118c63002c7a524844c30a624ae3c363ef912ba46252: Status 404 returned error can't find the container with id 93afde8159b211141399118c63002c7a524844c30a624ae3c363ef912ba46252
Sep 11 21:17:35.469861 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.469835    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5851258e_a72c_48ff_86db_af65f9b5f70e.slice/crio-5405a6c62739bff6afe37817cab321e362c8880c92ac2f25b61e636b606e904c WatchSource:0}: Error finding container 5405a6c62739bff6afe37817cab321e362c8880c92ac2f25b61e636b606e904c: Status 404 returned error can't find the container with id 5405a6c62739bff6afe37817cab321e362c8880c92ac2f25b61e636b606e904c
Sep 11 21:17:35.494216 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.494194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf250077_d04a_4193_8ab7_758369c6af18.slice/crio-54505496ae030ebf1367c99b5f4c768b9d3fa3190d4ec05d65e317fc67131e36 WatchSource:0}: Error finding container 54505496ae030ebf1367c99b5f4c768b9d3fa3190d4ec05d65e317fc67131e36: Status 404 returned error can't find the container with id 54505496ae030ebf1367c99b5f4c768b9d3fa3190d4ec05d65e317fc67131e36
Sep 11 21:17:35.494932 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.494879    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf02f91_8277_4937_a4cf_eda0b2e0c0d8.slice/crio-fd0d3ced2560b2d389719b97140b6879478191ea00f39e36b548181a17ff0c10 WatchSource:0}: Error finding container fd0d3ced2560b2d389719b97140b6879478191ea00f39e36b548181a17ff0c10: Status 404 returned error can't find the container with id fd0d3ced2560b2d389719b97140b6879478191ea00f39e36b548181a17ff0c10
Sep 11 21:17:35.496081 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.495927    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5ab34a_c777_443c_b624_4726afafe05d.slice/crio-a9533c17bb3eb4aa6c83a23d188d95bfac63fcfc985ca094077ad26bc6c81df2 WatchSource:0}: Error finding container a9533c17bb3eb4aa6c83a23d188d95bfac63fcfc985ca094077ad26bc6c81df2: Status 404 returned error can't find the container with id a9533c17bb3eb4aa6c83a23d188d95bfac63fcfc985ca094077ad26bc6c81df2
Sep 11 21:17:35.496400 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.496340    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5747e3b_6c03_4944_a051_520a9910f76e.slice/crio-6b71db2c2c0ff97a419958a5606afe1721b9d8e93fbc3146ac0eead9c45eedf3 WatchSource:0}: Error finding container 6b71db2c2c0ff97a419958a5606afe1721b9d8e93fbc3146ac0eead9c45eedf3: Status 404 returned error can't find the container with id 6b71db2c2c0ff97a419958a5606afe1721b9d8e93fbc3146ac0eead9c45eedf3
Sep 11 21:17:35.497282 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:17:35.497261    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a51e6e_8e64_4ae8_a1c4_9207f0322e19.slice/crio-c984e0cb76a8dce5fe56e5b2368a39c3e7bcb34e0e97ea25bea8c7b14ce18525 WatchSource:0}: Error finding container c984e0cb76a8dce5fe56e5b2368a39c3e7bcb34e0e97ea25bea8c7b14ce18525: Status 404 returned error can't find the container with id c984e0cb76a8dce5fe56e5b2368a39c3e7bcb34e0e97ea25bea8c7b14ce18525
Sep 11 21:17:35.911435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:35.911140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:35.911570 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.911315    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:35.911570 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:35.911565    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:37.911543125 +0000 UTC m=+6.033875026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.012576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.012620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012737    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012740    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012760    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012772    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012796    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:17:38.01277915 +0000 UTC m=+6.135111071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:36.012870 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.012825    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:38.012804535 +0000 UTC m=+6.135136435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:36.341235 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.341109    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:12:34 +0000 UTC" deadline="2028-05-14 02:18:04.976556225 +0000 UTC"
Sep 11 21:17:36.341235 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.341193    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14645h0m28.635367767s"
Sep 11 21:17:36.380641 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.377794    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:36.380641 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.377873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:36.380641 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.377976    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:36.380641 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:36.380582    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:36.389811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.389782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hbc5" event={"ID":"a5747e3b-6c03-4944-a051-520a9910f76e","Type":"ContainerStarted","Data":"6b71db2c2c0ff97a419958a5606afe1721b9d8e93fbc3146ac0eead9c45eedf3"}
Sep 11 21:17:36.393961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.393933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mmbt" event={"ID":"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8","Type":"ContainerStarted","Data":"fd0d3ced2560b2d389719b97140b6879478191ea00f39e36b548181a17ff0c10"}
Sep 11 21:17:36.396747 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.396719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j4mfs" event={"ID":"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19","Type":"ContainerStarted","Data":"c984e0cb76a8dce5fe56e5b2368a39c3e7bcb34e0e97ea25bea8c7b14ce18525"}
Sep 11 21:17:36.398374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.398348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr" event={"ID":"dc5ab34a-c777-443c-b624-4726afafe05d","Type":"ContainerStarted","Data":"a9533c17bb3eb4aa6c83a23d188d95bfac63fcfc985ca094077ad26bc6c81df2"}
Sep 11 21:17:36.401593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.401562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mc442" event={"ID":"df250077-d04a-4193-8ab7-758369c6af18","Type":"ContainerStarted","Data":"54505496ae030ebf1367c99b5f4c768b9d3fa3190d4ec05d65e317fc67131e36"}
Sep 11 21:17:36.410429 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.410404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdwrk" event={"ID":"03b3e290-3fd0-4911-9755-76b268887c68","Type":"ContainerStarted","Data":"74231cbc671c3c10b0ad7a077e9030271b3cb36113c6d8082d7320017371e32e"}
Sep 11 21:17:36.420673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.420648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerStarted","Data":"5405a6c62739bff6afe37817cab321e362c8880c92ac2f25b61e636b606e904c"}
Sep 11 21:17:36.429076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.429051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7sqxn" event={"ID":"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02","Type":"ContainerStarted","Data":"397056f893999fa93f8e5e0daf0e6a67ca8f935741851ee0d4f433e7a08eb84c"}
Sep 11 21:17:36.438385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.438360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"93afde8159b211141399118c63002c7a524844c30a624ae3c363ef912ba46252"}
Sep 11 21:17:36.443911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.443885    2580 generic.go:356] "Generic (PLEG): container finished" podID="f90cc286af7f28dc7b2553d2cebe6176" containerID="96ae161eeb9a9b70d3652c9fca7b8dc2a572ae7c31213260b6f9100ef09b59e5" exitCode=0
Sep 11 21:17:36.444007 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.443960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal" event={"ID":"f90cc286af7f28dc7b2553d2cebe6176","Type":"ContainerDied","Data":"96ae161eeb9a9b70d3652c9fca7b8dc2a572ae7c31213260b6f9100ef09b59e5"}
Sep 11 21:17:36.449741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:36.449718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal" event={"ID":"a7628f9440b1b266be07ea0dcc5e5880","Type":"ContainerStarted","Data":"afd17f6d78e56a99aaa33d04fea948d33f687a7b706d5986ea3c733c98340394"}
Sep 11 21:17:37.377994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:37.377964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:37.378617 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:37.378100    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:37.470587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:37.470554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal" event={"ID":"f90cc286af7f28dc7b2553d2cebe6176","Type":"ContainerStarted","Data":"efd1d6992f8f97b688eb4f59eef64982256f96afdffc6c4e37b7b08610ebddbc"}
Sep 11 21:17:37.505910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:37.505852    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-133.ec2.internal" podStartSLOduration=4.505830916 podStartE2EDuration="4.505830916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:36.489434732 +0000 UTC m=+4.611766655" watchObservedRunningTime="2026-09-11 21:17:37.505830916 +0000 UTC m=+5.628162839"
Sep 11 21:17:37.931411 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:37.931366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:37.931603 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:37.931520    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:37.931603 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:37.931581    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:41.931563663 +0000 UTC m=+10.053895564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:38.032101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:38.032171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032283    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032342    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:17:42.0323226 +0000 UTC m=+10.154654504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032746    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032764    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032776    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:38.032851 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.032817    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:42.032804139 +0000 UTC m=+10.155136043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:38.379952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:38.379876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:38.380394 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.380000    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:38.380394 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:38.380374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:38.380616 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:38.380545    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:39.377404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:39.377374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:39.377605 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:39.377522    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:40.379760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:40.378810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:40.379760 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:40.378931    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:40.379760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:40.379578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:40.379760 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:40.379684    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:41.377696 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:41.377649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:41.378031 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:41.377987    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:41.970068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:41.970021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:41.970451 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:41.970219    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:41.970451 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:41.970281    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:49.970262672 +0000 UTC m=+18.092594577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:42.070968 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:42.070933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:42.071155 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:42.070977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:42.071155 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071117    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:42.071155 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071138    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:42.071287 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071163    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:42.071287 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071177    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:42.071287 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071187    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:17:50.07116843 +0000 UTC m=+18.193500344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:42.071287 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.071224    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:17:50.071212578 +0000 UTC m=+18.193544495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:42.378094 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:42.378016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:42.378250 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.378099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:42.378250 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:42.378204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:42.378363 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:42.378313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:43.376969 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:43.376938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:43.377573 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:43.377081    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:44.379948 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:44.379917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:44.380351 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:44.380040    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:44.380405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:44.380376    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:44.380480 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:44.380458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:45.377403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:45.377370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:45.377581 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:45.377528    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:46.377307 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:46.377225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:46.377731 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:46.377355    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:46.377731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:46.377420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:46.377731 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:46.377565    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:47.377492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:47.377452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:47.377997 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:47.377614    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:48.377202 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:48.377165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:48.377202 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:48.377192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:48.377448 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:48.377303    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:48.377527 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:48.377450    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:49.377583 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:49.377546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:49.378031 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:49.377684    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:50.029231 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:50.029144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:50.029377 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.029322    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:50.029432 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.029386    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:06.029370643 +0000 UTC m=+34.151702542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:17:50.130102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:50.130063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:50.130102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:50.130109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130212    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130227    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130252    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130268    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130277    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:18:06.13025918 +0000 UTC m=+34.252591078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:17:50.130339 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.130318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:06.130301354 +0000 UTC m=+34.252633285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:17:50.377314 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:50.377232    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:50.377473 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.377352    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:50.377673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:50.377657    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:50.378024 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:50.377736    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:51.377247 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:51.377214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:51.377425 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:51.377345    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:52.377688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:52.377654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:52.378142 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:52.377757    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:52.378142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:52.377805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:52.378142 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:52.377942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:53.377463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:53.377277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:53.377463 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:53.377416    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:54.377715 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.377257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:54.378260 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.377263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:54.378260 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:54.377824    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:54.378260 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:54.377886    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:54.505599 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.505520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j4mfs" event={"ID":"c2a51e6e-8e64-4ae8-a1c4-9207f0322e19","Type":"ContainerStarted","Data":"f5e8acdbe60646ae898fd823745644675456aada4478d074d5be8d7384f93c4d"}
Sep 11 21:17:54.507047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.507022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr" event={"ID":"dc5ab34a-c777-443c-b624-4726afafe05d","Type":"ContainerStarted","Data":"767c680f19bb8598d8ddd9e96417352a7c90c8eef4b153b3a59570f36feb88f8"}
Sep 11 21:17:54.508538 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.508494    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mc442" event={"ID":"df250077-d04a-4193-8ab7-758369c6af18","Type":"ContainerStarted","Data":"6293bbd6e39d31dd1d87fb8f99ec9e74174f07a9a4a7936e8539e665c74f1512"}
Sep 11 21:17:54.510085 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.510060    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="5525b6d711fd564ff203d33f8cb0e78b2ecc60f06770163df117831177ec8c04" exitCode=0
Sep 11 21:17:54.510185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.510139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"5525b6d711fd564ff203d33f8cb0e78b2ecc60f06770163df117831177ec8c04"}
Sep 11 21:17:54.511676 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.511658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7sqxn" event={"ID":"4b8ec2fb-6130-4f3e-adda-0a8fe35bad02","Type":"ContainerStarted","Data":"3e42a7e048d3bae6b41de7b98d49285f65813d9c943f23d6db425483c34c3929"}
Sep 11 21:17:54.514659 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"5f6d2aef3f971fdc89828c2d11e80897836c64c55f50e2d176123cebc005a495"}
Sep 11 21:17:54.514761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"43ad26ebaa4a1305e53170c0aa7f934543636467c4054386ccbf79af202284bc"}
Sep 11 21:17:54.514761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"9d21713733c2c764d48cbf8ec1f7427a06dd9c6671608a577f6c63a968326e0f"}
Sep 11 21:17:54.514846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"7e512e3101297f9eabbaa4771b35e5fe5c5234319d37032906066a358f6d9b30"}
Sep 11 21:17:54.514846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"89cc8fd0379789e29f8d3f2d675c61be2825c596fa3c2c77bfde87623c39801f"}
Sep 11 21:17:54.514846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.514827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"614b78a86fde40b99863d8dfc6121f1006d3925aabe9412d7d03df5f420e0fd0"}
Sep 11 21:17:54.516154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.516134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hbc5" event={"ID":"a5747e3b-6c03-4944-a051-520a9910f76e","Type":"ContainerStarted","Data":"5b5059476bacb4610785317cc4f2e3eee4e59f1697d8388a48fead39f2075e7a"}
Sep 11 21:17:54.517561 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.517541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mmbt" event={"ID":"9bf02f91-8277-4937-a4cf-eda0b2e0c0d8","Type":"ContainerStarted","Data":"ad54e1dc55f8a454462f166e8439217100d6af9cbc858217c929dd55b207197b"}
Sep 11 21:17:54.520784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.520750    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-133.ec2.internal" podStartSLOduration=21.520739159 podStartE2EDuration="21.520739159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:37.506748825 +0000 UTC m=+5.629080747" watchObservedRunningTime="2026-09-11 21:17:54.520739159 +0000 UTC m=+22.643071079"
Sep 11 21:17:54.521169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.521147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j4mfs" podStartSLOduration=4.470980284 podStartE2EDuration="22.521142764s" totalImagesPullingTime="18.05016248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.520734825 +0000 UTC m=+22.643066746" watchObservedRunningTime="2026-09-11 21:17:54.521142764 +0000 UTC m=+22.643474684"
Sep 11 21:17:54.548206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.548154    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7sqxn" podStartSLOduration=4.473375823 podStartE2EDuration="22.548135493s" totalImagesPullingTime="18.07475967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.547204987 +0000 UTC m=+22.669536909" watchObservedRunningTime="2026-09-11 21:17:54.548135493 +0000 UTC m=+22.670467418"
Sep 11 21:17:54.591761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.591705    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6hbc5" podStartSLOduration=4.486413231 podStartE2EDuration="22.591685182s" totalImagesPullingTime="18.105271951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.591463757 +0000 UTC m=+22.713795678" watchObservedRunningTime="2026-09-11 21:17:54.591685182 +0000 UTC m=+22.714017104"
Sep 11 21:17:54.592310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.592282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6mmbt" podStartSLOduration=4.542129592 podStartE2EDuration="22.592269821s" totalImagesPullingTime="18.050140229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.56622001 +0000 UTC m=+22.688551946" watchObservedRunningTime="2026-09-11 21:17:54.592269821 +0000 UTC m=+22.714601746"
Sep 11 21:17:54.645158 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:54.645106    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mc442" podStartSLOduration=8.389787971 podStartE2EDuration="22.645092054s" totalImagesPullingTime="14.255304083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.644707889 +0000 UTC m=+22.767039809" watchObservedRunningTime="2026-09-11 21:17:54.645092054 +0000 UTC m=+22.767423974"
Sep 11 21:17:55.179803 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.179769    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:17:55.350760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.350597    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:17:55.179800952Z","UUID":"8b8f6626-635c-4723-83a4-1d30caa9e1b1","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:17:55.352228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.352206    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:17:55.352335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.352243    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:17:55.377138 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.377078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:55.377254 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:55.377185    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:55.522442 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.522404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr" event={"ID":"dc5ab34a-c777-443c-b624-4726afafe05d","Type":"ContainerStarted","Data":"49e2cfc7e45f5bfea0e9e09c225e2780516200f8fef58d5964b3bb549a413aa2"}
Sep 11 21:17:55.524586 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.524055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wdwrk" event={"ID":"03b3e290-3fd0-4911-9755-76b268887c68","Type":"ContainerStarted","Data":"165ba48187fd00a6590a189aef706e0a393bdd015873760e0cce96fca10fda64"}
Sep 11 21:17:55.539930 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:55.539886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wdwrk" podStartSLOduration=5.476336838 podStartE2EDuration="23.539867363s" totalImagesPullingTime="18.063530525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:55.539789034 +0000 UTC m=+23.662120956" watchObservedRunningTime="2026-09-11 21:17:55.539867363 +0000 UTC m=+23.662199286"
Sep 11 21:17:56.379199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:56.379169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:56.379315 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:56.379208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:56.379315 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:56.379289    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:56.379684 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:56.379659    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:56.529550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:56.529445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr" event={"ID":"dc5ab34a-c777-443c-b624-4726afafe05d","Type":"ContainerStarted","Data":"b57786ad8f17822fd60389b3c5cf734ed21c11367ebfc7cf3c796d64038dddc1"}
Sep 11 21:17:56.533604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:56.533556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"1b7d720b0043dc16c2167e59ebb1422fd37784e33ac800a5297cd9a0b7e0630c"}
Sep 11 21:17:56.549722 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:56.549673    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h89jr" podStartSLOduration=4.16361418 podStartE2EDuration="24.549651721s" totalImagesPullingTime="20.386037541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:56.549224007 +0000 UTC m=+24.671555933" watchObservedRunningTime="2026-09-11 21:17:56.549651721 +0000 UTC m=+24.671983643"
Sep 11 21:17:57.377611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:57.377581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:57.377790 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:57.377702    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:58.302900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.302650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:58.303618 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.303594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:58.378049 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.378007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:17:58.378216 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:58.378127    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:17:58.378216 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.378198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:17:58.378317 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:58.378293    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:17:58.537602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.537571    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:58.538107 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:58.538076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mc442"
Sep 11 21:17:59.377719 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.377696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:17:59.378061 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:17:59.377810    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:17:59.540793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.540768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerStarted","Data":"ad20d8336436b2d2c2b411364fd8c0ac3835c82910897b2297aa7e8776e3868b"}
Sep 11 21:17:59.544650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.544625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" event={"ID":"ca456b7e-4441-4ed3-9308-4ec9e34c45aa","Type":"ContainerStarted","Data":"f091db8fcac971d07f1d77e78249f66d16c80333db0b5689d053d204d01bb7b5"}
Sep 11 21:17:59.544960 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.544936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:59.545119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.545019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:59.560435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.560410    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:59.561670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.561654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:17:59.595204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:17:59.595151    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq" podStartSLOduration=9.45870932 podStartE2EDuration="27.59513712s" totalImagesPullingTime="18.1364278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:59.592710452 +0000 UTC m=+27.715042375" watchObservedRunningTime="2026-09-11 21:17:59.59513712 +0000 UTC m=+27.717469041"
Sep 11 21:18:00.378022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.377985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:00.378453 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:00.378125    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:18:00.378453 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.378181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:00.378453 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:00.378269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:18:00.548724 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.548695    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="ad20d8336436b2d2c2b411364fd8c0ac3835c82910897b2297aa7e8776e3868b" exitCode=0
Sep 11 21:18:00.548974 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.548787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"ad20d8336436b2d2c2b411364fd8c0ac3835c82910897b2297aa7e8776e3868b"}
Sep 11 21:18:00.548974 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.548882    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 21:18:00.616153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:00.615993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:18:01.061634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.061555    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7n7b"]
Sep 11 21:18:01.061795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.061646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:01.061795 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:01.061729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:18:01.064486 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.064461    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc88x"]
Sep 11 21:18:01.064609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.064577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:01.064679 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:01.064662    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:18:01.065256 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.065237    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-96jdf"]
Sep 11 21:18:01.065329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:01.065316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:01.065662 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:01.065399    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:18:02.380227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:02.380197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:02.381061 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:02.380351    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:18:02.381061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:02.380628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:02.381061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:02.380644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:02.381061 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:02.380769    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:18:02.381061 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:02.380885    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:18:02.555904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:02.555861    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="8dd5de17157d672973e17a174d5e453127b50685846118763c9fcff94e2d6d1c" exitCode=0
Sep 11 21:18:02.556056 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:02.555929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"8dd5de17157d672973e17a174d5e453127b50685846118763c9fcff94e2d6d1c"}
Sep 11 21:18:04.378128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:04.377867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:04.378488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:04.377873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:04.378488 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:04.378226    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:18:04.378488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:04.377904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:04.378488 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:04.378297    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:18:04.378488 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:04.378352    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:18:04.562612 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:04.562582    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="d72ca2dc981a4ada6bfa348fcb69c0f406f2b7facd6643d3dea27e3cab16112e" exitCode=0
Sep 11 21:18:04.562771 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:04.562633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"d72ca2dc981a4ada6bfa348fcb69c0f406f2b7facd6643d3dea27e3cab16112e"}
Sep 11 21:18:06.046559 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.046521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:06.047102 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.046654    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:18:06.047102 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.046723    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret podName:75b7b403-2cec-4496-b4d5-11ac365f0305 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:38.046704102 +0000 UTC m=+66.169036022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret") pod "global-pull-secret-syncer-p7n7b" (UID: "75b7b403-2cec-4496-b4d5-11ac365f0305") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:18:06.147215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.147181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:06.147379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.147226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:06.147379 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147336    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:18:06.147379 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147360    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:18:06.147379 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147366    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:18:06.147379 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147373    2580 projected.go:196] Error preparing data for projected volume kube-api-access-dwd96 for pod openshift-network-diagnostics/network-check-target-96jdf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:18:06.147655 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147422    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs podName:fdc27677-ab18-4b21-b0d2-994d107b7a9a nodeName:}" failed. No retries permitted until 2026-09-11 21:18:38.147403887 +0000 UTC m=+66.269735787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs") pod "network-metrics-daemon-tc88x" (UID: "fdc27677-ab18-4b21-b0d2-994d107b7a9a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:18:06.147655 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.147442    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96 podName:d91fa252-fe02-452c-8282-7106d86c01b0 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:38.147431183 +0000 UTC m=+66.269763085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dwd96" (UniqueName: "kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96") pod "network-check-target-96jdf" (UID: "d91fa252-fe02-452c-8282-7106d86c01b0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:18:06.377883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.377796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:06.377883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.377832    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:06.378096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:06.377798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:06.378096 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.377926    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7n7b" podUID="75b7b403-2cec-4496-b4d5-11ac365f0305"
Sep 11 21:18:06.378096 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.378000    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-96jdf" podUID="d91fa252-fe02-452c-8282-7106d86c01b0"
Sep 11 21:18:06.378096 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:06.378077    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tc88x" podUID="fdc27677-ab18-4b21-b0d2-994d107b7a9a"
Sep 11 21:18:07.148393 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.148363    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-133.ec2.internal" event="NodeReady"
Sep 11 21:18:07.148866 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.148539    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:18:07.217721 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.217682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q8s9c"]
Sep 11 21:18:07.222176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.222155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ht7j2"]
Sep 11 21:18:07.222349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.222264    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.225296 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.225274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.225971 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.225946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:18:07.226063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.225957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:18:07.226126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.225996    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:18:07.226126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.226076    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:18:07.226489 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.226472    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qrmzx\""
Sep 11 21:18:07.228099 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.228085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hdgmr\""
Sep 11 21:18:07.228367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.228352    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:18:07.228525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.228470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:18:07.240215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.240190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ht7j2"]
Sep 11 21:18:07.240772 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.240741    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pzf2m"]
Sep 11 21:18:07.243687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.243655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8s9c"]
Sep 11 21:18:07.243776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.243748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.246549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.246528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:18:07.247095 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.247075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:18:07.247246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.247229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ld6cs\""
Sep 11 21:18:07.247804 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.247773    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:18:07.258980 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.258947    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzf2m"]
Sep 11 21:18:07.355321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38eddb37-c2bc-4554-b33b-f38af8bac54f-cert\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.355321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfd6\" (UniqueName: \"kubernetes.io/projected/38eddb37-c2bc-4554-b33b-f38af8bac54f-kube-api-access-5vfd6\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.355321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4da237f-c80d-485a-a623-53480965fe1d-tmp-dir\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzrs\" (UniqueName: \"kubernetes.io/projected/b4da237f-c80d-485a-a623-53480965fe1d-kube-api-access-5jzrs\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f55297f1-2172-4f28-affa-157d70a431c1-crio-socket\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f55297f1-2172-4f28-affa-157d70a431c1-data-volume\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.355611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4da237f-c80d-485a-a623-53480965fe1d-config-volume\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.355903 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.355903 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.355669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdsj\" (UniqueName: \"kubernetes.io/projected/f55297f1-2172-4f28-affa-157d70a431c1-kube-api-access-wkdsj\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.376817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.376786    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mmbt_9bf02f91-8277-4937-a4cf-eda0b2e0c0d8/dns-node-resolver/0.log"
Sep 11 21:18:07.456287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdsj\" (UniqueName: \"kubernetes.io/projected/f55297f1-2172-4f28-affa-157d70a431c1-kube-api-access-wkdsj\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.456458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38eddb37-c2bc-4554-b33b-f38af8bac54f-cert\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.456458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfd6\" (UniqueName: \"kubernetes.io/projected/38eddb37-c2bc-4554-b33b-f38af8bac54f-kube-api-access-5vfd6\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.456458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4da237f-c80d-485a-a623-53480965fe1d-tmp-dir\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.456458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzrs\" (UniqueName: \"kubernetes.io/projected/b4da237f-c80d-485a-a623-53480965fe1d-kube-api-access-5jzrs\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.456725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.456857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f55297f1-2172-4f28-affa-157d70a431c1-crio-socket\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.456857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f55297f1-2172-4f28-affa-157d70a431c1-data-volume\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.456857 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.456790    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:18:07.456857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.456857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4da237f-c80d-485a-a623-53480965fe1d-config-volume\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.456868    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:07.956846925 +0000 UTC m=+36.079178840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : configmap "kube-rbac-proxy" not found
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.456966    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4da237f-c80d-485a-a623-53480965fe1d-tmp-dir\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.456997    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.457030    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:07.957021103 +0000 UTC m=+36.079353002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.457044    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.457082    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls podName:b4da237f-c80d-485a-a623-53480965fe1d nodeName:}" failed. No retries permitted until 2026-09-11 21:18:07.957066914 +0000 UTC m=+36.079398813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls") pod "dns-default-ht7j2" (UID: "b4da237f-c80d-485a-a623-53480965fe1d") : secret "dns-default-metrics-tls" not found
Sep 11 21:18:07.457120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.457087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f55297f1-2172-4f28-affa-157d70a431c1-data-volume\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.457377 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.457208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f55297f1-2172-4f28-affa-157d70a431c1-crio-socket\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.457656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.457640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4da237f-c80d-485a-a623-53480965fe1d-config-volume\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.461439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.461415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38eddb37-c2bc-4554-b33b-f38af8bac54f-cert\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.479432 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.479405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzrs\" (UniqueName: \"kubernetes.io/projected/b4da237f-c80d-485a-a623-53480965fe1d-kube-api-access-5jzrs\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.480401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.480372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdsj\" (UniqueName: \"kubernetes.io/projected/f55297f1-2172-4f28-affa-157d70a431c1-kube-api-access-wkdsj\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.482817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.482792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfd6\" (UniqueName: \"kubernetes.io/projected/38eddb37-c2bc-4554-b33b-f38af8bac54f-kube-api-access-5vfd6\") pod \"ingress-canary-pzf2m\" (UID: \"38eddb37-c2bc-4554-b33b-f38af8bac54f\") " pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.556821 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.556790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pzf2m"
Sep 11 21:18:07.961333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.961299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.961549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.961351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:07.961549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.961386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:07.961549 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.961495    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:07.961549 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.961531    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:18:07.961760 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.961580    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:08.961562748 +0000 UTC m=+37.083894648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:07.961760 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:07.961599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls podName:b4da237f-c80d-485a-a623-53480965fe1d nodeName:}" failed. No retries permitted until 2026-09-11 21:18:08.961590004 +0000 UTC m=+37.083921902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls") pod "dns-default-ht7j2" (UID: "b4da237f-c80d-485a-a623-53480965fe1d") : secret "dns-default-metrics-tls" not found
Sep 11 21:18:07.961916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:07.961892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f55297f1-2172-4f28-affa-157d70a431c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:08.337027 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.336947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j4mfs_c2a51e6e-8e64-4ae8-a1c4-9207f0322e19/node-ca/0.log"
Sep 11 21:18:08.377794 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.377758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:08.377945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.377815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:08.377945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.377828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382019    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vqqz8\""
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vp72k\""
Sep 11 21:18:08.382483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.382347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:18:08.616455 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.615609    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-v64ps"]
Sep 11 21:18:08.619676 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.619656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.622238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.622207    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 21:18:08.622596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.622581    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 21:18:08.623391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.623368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 21:18:08.623477 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.623368    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 21:18:08.623477 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.623374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p2wdn\""
Sep 11 21:18:08.627974 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.627954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-v64ps"]
Sep 11 21:18:08.767196 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.767162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-key\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.767322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.767282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-cabundle\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.767322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.767317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5hl\" (UniqueName: \"kubernetes.io/projected/90698091-79b4-46e8-9c88-2dbd75c33c95-kube-api-access-gd5hl\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.801086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.800944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pzf2m"]
Sep 11 21:18:08.828245 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:08.828221    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eddb37_c2bc_4554_b33b_f38af8bac54f.slice/crio-f994c8bc1389b5247168cf2b0638d392ab773f9b7f11931fb6adf3ab5e43d05a WatchSource:0}: Error finding container f994c8bc1389b5247168cf2b0638d392ab773f9b7f11931fb6adf3ab5e43d05a: Status 404 returned error can't find the container with id f994c8bc1389b5247168cf2b0638d392ab773f9b7f11931fb6adf3ab5e43d05a
Sep 11 21:18:08.868068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.868011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-cabundle\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.868068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.868047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5hl\" (UniqueName: \"kubernetes.io/projected/90698091-79b4-46e8-9c88-2dbd75c33c95-kube-api-access-gd5hl\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.868222 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.868096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-key\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.868678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.868658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-cabundle\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.871149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.871128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/90698091-79b4-46e8-9c88-2dbd75c33c95-signing-key\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.877985 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.877968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5hl\" (UniqueName: \"kubernetes.io/projected/90698091-79b4-46e8-9c88-2dbd75c33c95-kube-api-access-gd5hl\") pod \"service-ca-568869fdc5-v64ps\" (UID: \"90698091-79b4-46e8-9c88-2dbd75c33c95\") " pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.930967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.930944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-v64ps"
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.969482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:08.969562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:08.969597    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:08.969656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls podName:b4da237f-c80d-485a-a623-53480965fe1d nodeName:}" failed. No retries permitted until 2026-09-11 21:18:10.969641446 +0000 UTC m=+39.091973345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls") pod "dns-default-ht7j2" (UID: "b4da237f-c80d-485a-a623-53480965fe1d") : secret "dns-default-metrics-tls" not found
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:08.969679    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:08.969725 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:08.969731    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:10.969715507 +0000 UTC m=+39.092047424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:09.202616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:09.202548    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-v64ps"]
Sep 11 21:18:09.205732 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:09.205707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90698091_79b4_46e8_9c88_2dbd75c33c95.slice/crio-c20f67d51e0c646319b0b074572f1df89afd6fc63549d3bec967eb75e2834aef WatchSource:0}: Error finding container c20f67d51e0c646319b0b074572f1df89afd6fc63549d3bec967eb75e2834aef: Status 404 returned error can't find the container with id c20f67d51e0c646319b0b074572f1df89afd6fc63549d3bec967eb75e2834aef
Sep 11 21:18:09.576778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:09.576699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-v64ps" event={"ID":"90698091-79b4-46e8-9c88-2dbd75c33c95","Type":"ContainerStarted","Data":"c20f67d51e0c646319b0b074572f1df89afd6fc63549d3bec967eb75e2834aef"}
Sep 11 21:18:09.578907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:09.578873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzf2m" event={"ID":"38eddb37-c2bc-4554-b33b-f38af8bac54f","Type":"ContainerStarted","Data":"f994c8bc1389b5247168cf2b0638d392ab773f9b7f11931fb6adf3ab5e43d05a"}
Sep 11 21:18:09.581740 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:09.581711    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="25f9d201911dde838bf9f404bc06a9ae6393902f14117f218c6c4f11cd2168d0" exitCode=0
Sep 11 21:18:09.581845 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:09.581763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"25f9d201911dde838bf9f404bc06a9ae6393902f14117f218c6c4f11cd2168d0"}
Sep 11 21:18:10.588549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:10.588515    2580 generic.go:356] "Generic (PLEG): container finished" podID="5851258e-a72c-48ff-86db-af65f9b5f70e" containerID="01b99ce2db7edf7e33614fafc79f8a66bbc2cf888adbfce356c3155580f5169c" exitCode=0
Sep 11 21:18:10.588549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:10.588541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerDied","Data":"01b99ce2db7edf7e33614fafc79f8a66bbc2cf888adbfce356c3155580f5169c"}
Sep 11 21:18:10.989173 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:10.989092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:10.989173 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:10.989130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:10.989370 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:10.989244    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:10.989370 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:10.989255    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:18:10.989370 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:10.989316    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:14.989295032 +0000 UTC m=+43.111626969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:10.989370 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:10.989336    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls podName:b4da237f-c80d-485a-a623-53480965fe1d nodeName:}" failed. No retries permitted until 2026-09-11 21:18:14.989326463 +0000 UTC m=+43.111658361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls") pod "dns-default-ht7j2" (UID: "b4da237f-c80d-485a-a623-53480965fe1d") : secret "dns-default-metrics-tls" not found
Sep 11 21:18:12.596848 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.596753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-v64ps" event={"ID":"90698091-79b4-46e8-9c88-2dbd75c33c95","Type":"ContainerStarted","Data":"9ccdf7ed4593e6908ea8249346afe96eb878984c825bfa5cbeceee45c66e20b4"}
Sep 11 21:18:12.598403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.598379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pzf2m" event={"ID":"38eddb37-c2bc-4554-b33b-f38af8bac54f","Type":"ContainerStarted","Data":"ad4d68acece7f042ce323eaf415f72e7aa41edd3c3d9ac0719c5fa7236fe633e"}
Sep 11 21:18:12.601853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.601824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" event={"ID":"5851258e-a72c-48ff-86db-af65f9b5f70e","Type":"ContainerStarted","Data":"845c13dec956ebdb35afaa1272c463c12029a434a7dac8e2670e14504272846b"}
Sep 11 21:18:12.634687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.634652    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-v64ps" podStartSLOduration=1.554316707 podStartE2EDuration="4.634640043s" totalImagesPullingTime="3.080323336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:12.634209139 +0000 UTC m=+40.756541059" watchObservedRunningTime="2026-09-11 21:18:12.634640043 +0000 UTC m=+40.756971963"
Sep 11 21:18:12.664038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.663997    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pzf2m" podStartSLOduration=2.211141836 podStartE2EDuration="5.663984934s" totalImagesPullingTime="3.452843098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:12.662109648 +0000 UTC m=+40.784441568" watchObservedRunningTime="2026-09-11 21:18:12.663984934 +0000 UTC m=+40.786316854"
Sep 11 21:18:12.702224 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:12.702190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gx6bl" podStartSLOduration=11.590046978 podStartE2EDuration="40.702178093s" totalImagesPullingTime="29.112131115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:12.7015983 +0000 UTC m=+40.823930245" watchObservedRunningTime="2026-09-11 21:18:12.702178093 +0000 UTC m=+40.824510014"
Sep 11 21:18:15.024180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:15.024145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:15.024180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:15.024181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:15.024623 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:15.024267    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:15.024623 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:15.024280    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:18:15.024623 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:15.024318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls podName:f55297f1-2172-4f28-affa-157d70a431c1 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:23.02430447 +0000 UTC m=+51.146636370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8s9c" (UID: "f55297f1-2172-4f28-affa-157d70a431c1") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:18:15.024623 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:15.024330    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls podName:b4da237f-c80d-485a-a623-53480965fe1d nodeName:}" failed. No retries permitted until 2026-09-11 21:18:23.024324641 +0000 UTC m=+51.146656539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls") pod "dns-default-ht7j2" (UID: "b4da237f-c80d-485a-a623-53480965fe1d") : secret "dns-default-metrics-tls" not found
Sep 11 21:18:23.077263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.077225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:23.077263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.077269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:23.080253 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.080225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4da237f-c80d-485a-a623-53480965fe1d-metrics-tls\") pod \"dns-default-ht7j2\" (UID: \"b4da237f-c80d-485a-a623-53480965fe1d\") " pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:23.080379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.080252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f55297f1-2172-4f28-affa-157d70a431c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8s9c\" (UID: \"f55297f1-2172-4f28-affa-157d70a431c1\") " pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:23.134593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.134566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8s9c"
Sep 11 21:18:23.139732 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.139713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:23.430946 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.430873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8s9c"]
Sep 11 21:18:23.448246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.448225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ht7j2"]
Sep 11 21:18:23.450761 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:23.450736    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4da237f_c80d_485a_a623_53480965fe1d.slice/crio-1522577c730fe6f358cbc4c19b905a09e91802078e0d62eaf5c9ef8e372aedfc WatchSource:0}: Error finding container 1522577c730fe6f358cbc4c19b905a09e91802078e0d62eaf5c9ef8e372aedfc: Status 404 returned error can't find the container with id 1522577c730fe6f358cbc4c19b905a09e91802078e0d62eaf5c9ef8e372aedfc
Sep 11 21:18:23.629688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.629658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht7j2" event={"ID":"b4da237f-c80d-485a-a623-53480965fe1d","Type":"ContainerStarted","Data":"1522577c730fe6f358cbc4c19b905a09e91802078e0d62eaf5c9ef8e372aedfc"}
Sep 11 21:18:23.631210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.631186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8s9c" event={"ID":"f55297f1-2172-4f28-affa-157d70a431c1","Type":"ContainerStarted","Data":"3a46df24dc8a3b27afb8f86e6e1a8c08184c20f7afa43864eac1e09ee7b88c28"}
Sep 11 21:18:23.631296 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:23.631236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8s9c" event={"ID":"f55297f1-2172-4f28-affa-157d70a431c1","Type":"ContainerStarted","Data":"000a6ef77a22b31cabf144018105cd43fa9ce6d093f1c466b3f6303457d3eb09"}
Sep 11 21:18:25.638169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:25.637973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht7j2" event={"ID":"b4da237f-c80d-485a-a623-53480965fe1d","Type":"ContainerStarted","Data":"13df4d3283770bf9629c07b9f35ecf7dd636ef82cc13cc5fd115879822f94014"}
Sep 11 21:18:25.638169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:25.638151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:25.638680 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:25.638195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht7j2" event={"ID":"b4da237f-c80d-485a-a623-53480965fe1d","Type":"ContainerStarted","Data":"3f356b6cc6bf353344db724121ed7505987b674993d45cd6821eb05ee8c26116"}
Sep 11 21:18:25.640111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:25.640086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8s9c" event={"ID":"f55297f1-2172-4f28-affa-157d70a431c1","Type":"ContainerStarted","Data":"4077094f0e033e2318fbc66838d88ca4c126a09138ce9e1811c61ce57d9d01a2"}
Sep 11 21:18:25.656444 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:25.656395    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ht7j2" podStartSLOduration=16.783821982 podStartE2EDuration="18.656380351s" totalImagesPullingTime="1.872558369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:25.655335111 +0000 UTC m=+53.777667033" watchObservedRunningTime="2026-09-11 21:18:25.656380351 +0000 UTC m=+53.778712271"
Sep 11 21:18:26.645694 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:26.645660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8s9c" event={"ID":"f55297f1-2172-4f28-affa-157d70a431c1","Type":"ContainerStarted","Data":"f97600a1ca71a53ec488d536736cf1a49f83720409ed9ffb894c6b82d84c65e8"}
Sep 11 21:18:26.689279 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:26.689240    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q8s9c" podStartSLOduration=17.17918536 podStartE2EDuration="19.689226619s" totalImagesPullingTime="2.510041259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:18:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:26.68893033 +0000 UTC m=+54.811262250" watchObservedRunningTime="2026-09-11 21:18:26.689226619 +0000 UTC m=+54.811558540"
Sep 11 21:18:31.113177 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.113149    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"]
Sep 11 21:18:31.141629 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.141466    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"]
Sep 11 21:18:31.141803 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.141721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.147753 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.147723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4sjrb\""
Sep 11 21:18:31.147890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.147761    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:18:31.147890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.147795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:18:31.147890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.147731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 21:18:31.153193 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.153176    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:18:31.154785 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.154771    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"]
Sep 11 21:18:31.154875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.154863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"
Sep 11 21:18:31.158483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.158466    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-psssv\""
Sep 11 21:18:31.168720 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.168703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"]
Sep 11 21:18:31.168805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.168725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"]
Sep 11 21:18:31.168805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.168742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"]
Sep 11 21:18:31.168805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.168802    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.171371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.171349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 21:18:31.171371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.171361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 21:18:31.171523 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.171437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 21:18:31.171657 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.171644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 21:18:31.234055 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.234055 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.234195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxfs\" (UniqueName: \"kubernetes.io/projected/748acb56-1a80-406d-8ba7-7dbc38be275c-kube-api-access-rjxfs\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.234195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswxf\" (UniqueName: \"kubernetes.io/projected/0c124b94-6c81-44d9-b3ec-143bfd5c0ace-kube-api-access-wswxf\") pod \"network-check-source-6f8b768455-7sw87\" (UID: \"0c124b94-6c81-44d9-b3ec-143bfd5c0ace\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"
Sep 11 21:18:31.234195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.234289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg65\" (UniqueName: \"kubernetes.io/projected/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-kube-api-access-dkg65\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.234289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.234289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.234376 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.234301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/748acb56-1a80-406d-8ba7-7dbc38be275c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.274601 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.274575    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f578bb4fd-4dphg"]
Sep 11 21:18:31.294206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.294186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.296964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.296946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:18:31.297460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.297443    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:18:31.297786 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.297443    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xvbfg\""
Sep 11 21:18:31.297786 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.297760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f578bb4fd-4dphg"]
Sep 11 21:18:31.298150 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.298135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:18:31.305389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.305368    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:18:31.334966 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.334945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-trusted-ca\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.334975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.334996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-registry-certificates\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswxf\" (UniqueName: \"kubernetes.io/projected/0c124b94-6c81-44d9-b3ec-143bfd5c0ace-kube-api-access-wswxf\") pod \"network-check-source-6f8b768455-7sw87\" (UID: \"0c124b94-6c81-44d9-b3ec-143bfd5c0ace\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"
Sep 11 21:18:31.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96bx\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-kube-api-access-l96bx\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-image-registry-private-configuration\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg65\" (UniqueName: \"kubernetes.io/projected/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-kube-api-access-dkg65\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a388ba12-8b45-4705-a47b-b718205da999-ca-trust-extracted\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/748acb56-1a80-406d-8ba7-7dbc38be275c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-bound-sa-token\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxfs\" (UniqueName: \"kubernetes.io/projected/748acb56-1a80-406d-8ba7-7dbc38be275c-kube-api-access-rjxfs\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.335749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-registry-tls\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.335749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.335487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-installation-pull-secrets\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.336306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.336280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/748acb56-1a80-406d-8ba7-7dbc38be275c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.337936 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.337912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.338022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.337982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-ca\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.338085 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.338067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.338267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.338245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/748acb56-1a80-406d-8ba7-7dbc38be275c-hub\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.338312 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.338301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.354830 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.354809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg65\" (UniqueName: \"kubernetes.io/projected/3c37223f-19d8-4b2e-93b5-a185fe9d6f1f-kube-api-access-dkg65\") pod \"managed-serviceaccount-addon-agent-6c75774f4-jdrbj\" (UID: \"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.356583 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.356560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxfs\" (UniqueName: \"kubernetes.io/projected/748acb56-1a80-406d-8ba7-7dbc38be275c-kube-api-access-rjxfs\") pod \"cluster-proxy-proxy-agent-6bb8cdbdbc-j974j\" (UID: \"748acb56-1a80-406d-8ba7-7dbc38be275c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.357040 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.357019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswxf\" (UniqueName: \"kubernetes.io/projected/0c124b94-6c81-44d9-b3ec-143bfd5c0ace-kube-api-access-wswxf\") pod \"network-check-source-6f8b768455-7sw87\" (UID: \"0c124b94-6c81-44d9-b3ec-143bfd5c0ace\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"
Sep 11 21:18:31.436127 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-image-registry-private-configuration\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436127 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a388ba12-8b45-4705-a47b-b718205da999-ca-trust-extracted\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-bound-sa-token\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-registry-tls\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-installation-pull-secrets\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-trusted-ca\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-registry-certificates\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436544 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96bx\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-kube-api-access-l96bx\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.436544 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.436466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a388ba12-8b45-4705-a47b-b718205da999-ca-trust-extracted\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.437121 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.437095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-registry-certificates\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.437298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.437279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a388ba12-8b45-4705-a47b-b718205da999-trusted-ca\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.438382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.438362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-image-registry-private-configuration\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.438741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.438724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-registry-tls\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.438741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.438734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a388ba12-8b45-4705-a47b-b718205da999-installation-pull-secrets\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.448901 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.448883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96bx\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-kube-api-access-l96bx\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.454546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.454528    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a388ba12-8b45-4705-a47b-b718205da999-bound-sa-token\") pod \"image-registry-5f578bb4fd-4dphg\" (UID: \"a388ba12-8b45-4705-a47b-b718205da999\") " pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.461211 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.461197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"
Sep 11 21:18:31.467341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.467325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"
Sep 11 21:18:31.476543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.476530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"
Sep 11 21:18:31.609346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.609319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:31.770822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.770796    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj"]
Sep 11 21:18:31.773883 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:31.773859    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c37223f_19d8_4b2e_93b5_a185fe9d6f1f.slice/crio-7b34b161476b80c40407d83a0f90045394207a87a66836fcf3b7c97c7e160086 WatchSource:0}: Error finding container 7b34b161476b80c40407d83a0f90045394207a87a66836fcf3b7c97c7e160086: Status 404 returned error can't find the container with id 7b34b161476b80c40407d83a0f90045394207a87a66836fcf3b7c97c7e160086
Sep 11 21:18:31.790882 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.790857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-7sw87"]
Sep 11 21:18:31.794417 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:31.794395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c124b94_6c81_44d9_b3ec_143bfd5c0ace.slice/crio-177a7bf28281decd35a551a4ee881d3f9546a9e0756feca8497411da5bfdb844 WatchSource:0}: Error finding container 177a7bf28281decd35a551a4ee881d3f9546a9e0756feca8497411da5bfdb844: Status 404 returned error can't find the container with id 177a7bf28281decd35a551a4ee881d3f9546a9e0756feca8497411da5bfdb844
Sep 11 21:18:31.815239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.815219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j"]
Sep 11 21:18:31.829012 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:31.828982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod748acb56_1a80_406d_8ba7_7dbc38be275c.slice/crio-ebc2058549a4fb9a260903ea3cfc192a5765a4234e7b72858bc8e5d3d86f8f8f WatchSource:0}: Error finding container ebc2058549a4fb9a260903ea3cfc192a5765a4234e7b72858bc8e5d3d86f8f8f: Status 404 returned error can't find the container with id ebc2058549a4fb9a260903ea3cfc192a5765a4234e7b72858bc8e5d3d86f8f8f
Sep 11 21:18:31.907863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:31.907838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f578bb4fd-4dphg"]
Sep 11 21:18:31.910264 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:31.910246    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda388ba12_8b45_4705_a47b_b718205da999.slice/crio-ebd18af670acbc4b3ca34b7edddabab88f6f0175d182fe811582d0a6a9052566 WatchSource:0}: Error finding container ebd18af670acbc4b3ca34b7edddabab88f6f0175d182fe811582d0a6a9052566: Status 404 returned error can't find the container with id ebd18af670acbc4b3ca34b7edddabab88f6f0175d182fe811582d0a6a9052566
Sep 11 21:18:32.574645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.573644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fbxcq"
Sep 11 21:18:32.668862 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.668781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j" event={"ID":"748acb56-1a80-406d-8ba7-7dbc38be275c","Type":"ContainerStarted","Data":"ebc2058549a4fb9a260903ea3cfc192a5765a4234e7b72858bc8e5d3d86f8f8f"}
Sep 11 21:18:32.672705 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.672671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87" event={"ID":"0c124b94-6c81-44d9-b3ec-143bfd5c0ace","Type":"ContainerStarted","Data":"177a7bf28281decd35a551a4ee881d3f9546a9e0756feca8497411da5bfdb844"}
Sep 11 21:18:32.675899 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.675868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg" event={"ID":"a388ba12-8b45-4705-a47b-b718205da999","Type":"ContainerStarted","Data":"47d033394b373b4eb8ac8a8afca80c330bacc371ec27f5bfb3f7d3b70af9a474"}
Sep 11 21:18:32.676008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.675920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg" event={"ID":"a388ba12-8b45-4705-a47b-b718205da999","Type":"ContainerStarted","Data":"ebd18af670acbc4b3ca34b7edddabab88f6f0175d182fe811582d0a6a9052566"}
Sep 11 21:18:32.676687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.676667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:18:32.678739 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.678710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj" event={"ID":"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f","Type":"ContainerStarted","Data":"7b34b161476b80c40407d83a0f90045394207a87a66836fcf3b7c97c7e160086"}
Sep 11 21:18:32.702018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:32.701960    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg" podStartSLOduration=1.701939935 podStartE2EDuration="1.701939935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:32.69982783 +0000 UTC m=+60.822159773" watchObservedRunningTime="2026-09-11 21:18:32.701939935 +0000 UTC m=+60.824271856"
Sep 11 21:18:35.647770 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:35.647740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ht7j2"
Sep 11 21:18:37.701184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.701152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj" event={"ID":"3c37223f-19d8-4b2e-93b5-a185fe9d6f1f","Type":"ContainerStarted","Data":"3aed9db6cc813c3270b213655398eceb7c240f1402bcfb1c8f1913a9e467f703"}
Sep 11 21:18:37.703541 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.703496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j" event={"ID":"748acb56-1a80-406d-8ba7-7dbc38be275c","Type":"ContainerStarted","Data":"2f28b93a2a74d08438ea469ce5e13e3e5fbdd633f713a2291d71d4053bc201ba"}
Sep 11 21:18:37.703671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.703562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j" event={"ID":"748acb56-1a80-406d-8ba7-7dbc38be275c","Type":"ContainerStarted","Data":"ceb5255fae44ae6407c568cae4bf420992bb44554612d994b647ac40853f90cc"}
Sep 11 21:18:37.703671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.703585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j" event={"ID":"748acb56-1a80-406d-8ba7-7dbc38be275c","Type":"ContainerStarted","Data":"06d4545c169cb9fc89672ba09bb3d505dc82bc7a9dac0a9ef94432bc68eb6600"}
Sep 11 21:18:37.705075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.705053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87" event={"ID":"0c124b94-6c81-44d9-b3ec-143bfd5c0ace","Type":"ContainerStarted","Data":"14276cf524cce11c67fd0bb8bacd405734f097ca7eeecb2f6effb2a093b3a276"}
Sep 11 21:18:37.724018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.723969    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c75774f4-jdrbj" podStartSLOduration=1.363366448 podStartE2EDuration="6.723951263s" totalImagesPullingTime="5.360584815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:37.723484633 +0000 UTC m=+65.845816555" watchObservedRunningTime="2026-09-11 21:18:37.723951263 +0000 UTC m=+65.846283185"
Sep 11 21:18:37.746439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.746399    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bb8cdbdbc-j974j" podStartSLOduration=1.425594694 podStartE2EDuration="6.746385828s" totalImagesPullingTime="5.320791134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:37.74536118 +0000 UTC m=+65.867693100" watchObservedRunningTime="2026-09-11 21:18:37.746385828 +0000 UTC m=+65.868717748"
Sep 11 21:18:37.765981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:37.765943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-7sw87" podStartSLOduration=1.415768046 podStartE2EDuration="6.765930258s" totalImagesPullingTime="5.350162212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:37.765640024 +0000 UTC m=+65.887971957" watchObservedRunningTime="2026-09-11 21:18:37.765930258 +0000 UTC m=+65.888262173"
Sep 11 21:18:38.097684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.097644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:38.100706 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.100686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:18:38.110297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.110273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75b7b403-2cec-4496-b4d5-11ac365f0305-original-pull-secret\") pod \"global-pull-secret-syncer-p7n7b\" (UID: \"75b7b403-2cec-4496-b4d5-11ac365f0305\") " pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:38.198675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.198647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:38.198776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.198707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:38.201213 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.201193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd96\" (UniqueName: \"kubernetes.io/projected/d91fa252-fe02-452c-8282-7106d86c01b0-kube-api-access-dwd96\") pod \"network-check-target-96jdf\" (UID: \"d91fa252-fe02-452c-8282-7106d86c01b0\") " pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:38.201368 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.201350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:18:38.211571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.211550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc27677-ab18-4b21-b0d2-994d107b7a9a-metrics-certs\") pod \"network-metrics-daemon-tc88x\" (UID: \"fdc27677-ab18-4b21-b0d2-994d107b7a9a\") " pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:38.392320 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.392270    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vqqz8\""
Sep 11 21:18:38.397252 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.397239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7n7b"
Sep 11 21:18:38.400449 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.400435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tc88x"
Sep 11 21:18:38.405029 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.405014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vp72k\""
Sep 11 21:18:38.413202 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.413188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:38.688717 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.688654    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7n7b"]
Sep 11 21:18:38.691365 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:38.691342    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b7b403_2cec_4496_b4d5_11ac365f0305.slice/crio-0217efc215437730ede3264e14f766d7a69a95cc249499045e832d60d4425abf WatchSource:0}: Error finding container 0217efc215437730ede3264e14f766d7a69a95cc249499045e832d60d4425abf: Status 404 returned error can't find the container with id 0217efc215437730ede3264e14f766d7a69a95cc249499045e832d60d4425abf
Sep 11 21:18:38.709752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.709726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p7n7b" event={"ID":"75b7b403-2cec-4496-b4d5-11ac365f0305","Type":"ContainerStarted","Data":"0217efc215437730ede3264e14f766d7a69a95cc249499045e832d60d4425abf"}
Sep 11 21:18:38.724591 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.724571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tc88x"]
Sep 11 21:18:38.727070 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:38.727041    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc27677_ab18_4b21_b0d2_994d107b7a9a.slice/crio-6702548a4f0588b76a84b7ceaa85ffdb2da070decf69b5c38b8c444004dcffed WatchSource:0}: Error finding container 6702548a4f0588b76a84b7ceaa85ffdb2da070decf69b5c38b8c444004dcffed: Status 404 returned error can't find the container with id 6702548a4f0588b76a84b7ceaa85ffdb2da070decf69b5c38b8c444004dcffed
Sep 11 21:18:38.766897 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:38.766876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-96jdf"]
Sep 11 21:18:38.769265 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:38.769243    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91fa252_fe02_452c_8282_7106d86c01b0.slice/crio-48f09f6e0cfbbef9b9694b062914018704271a62b3c3aaa8abb53c9da27b61a9 WatchSource:0}: Error finding container 48f09f6e0cfbbef9b9694b062914018704271a62b3c3aaa8abb53c9da27b61a9: Status 404 returned error can't find the container with id 48f09f6e0cfbbef9b9694b062914018704271a62b3c3aaa8abb53c9da27b61a9
Sep 11 21:18:39.717301 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:39.717261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-96jdf" event={"ID":"d91fa252-fe02-452c-8282-7106d86c01b0","Type":"ContainerStarted","Data":"562753aa25bb0ad668796c9a6d7154750431b7df0c5cd8d95a298e0bfc618e17"}
Sep 11 21:18:39.717778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:39.717321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-96jdf" event={"ID":"d91fa252-fe02-452c-8282-7106d86c01b0","Type":"ContainerStarted","Data":"48f09f6e0cfbbef9b9694b062914018704271a62b3c3aaa8abb53c9da27b61a9"}
Sep 11 21:18:39.717778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:39.717347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:18:39.719876 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:39.719843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc88x" event={"ID":"fdc27677-ab18-4b21-b0d2-994d107b7a9a","Type":"ContainerStarted","Data":"6702548a4f0588b76a84b7ceaa85ffdb2da070decf69b5c38b8c444004dcffed"}
Sep 11 21:18:40.724817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:40.724779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc88x" event={"ID":"fdc27677-ab18-4b21-b0d2-994d107b7a9a","Type":"ContainerStarted","Data":"4cdc3cb0ac0951b5e6d3893324c373b0ba06ab6a7e2321127631e8d3ed380883"}
Sep 11 21:18:40.725212 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:40.724842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tc88x" event={"ID":"fdc27677-ab18-4b21-b0d2-994d107b7a9a","Type":"ContainerStarted","Data":"f4de439b44b117cc9811dc7c44df949a7859d289728488a96bba4e25085bd20d"}
Sep 11 21:18:40.757844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:40.757783    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-96jdf" podStartSLOduration=68.757762887 podStartE2EDuration="1m8.757762887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:39.733152383 +0000 UTC m=+67.855484305" watchObservedRunningTime="2026-09-11 21:18:40.757762887 +0000 UTC m=+68.880094810"
Sep 11 21:18:43.106437 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.106380    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tc88x" podStartSLOduration=69.892393116 podStartE2EDuration="1m11.106356132s" totalImagesPullingTime="1.213963016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:40.757958999 +0000 UTC m=+68.880290921" watchObservedRunningTime="2026-09-11 21:18:43.106356132 +0000 UTC m=+71.228688053"
Sep 11 21:18:43.107081 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.107048    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zr42f"]
Sep 11 21:18:43.109824 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.109797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.113788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.113766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:18:43.113788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.113783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:18:43.114947 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.114929    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:18:43.115141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.115126    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:18:43.115328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.115312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:18:43.115537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.115494    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4fgxn\""
Sep 11 21:18:43.115697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.115684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:18:43.234175 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-sys\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-metrics-client-ca\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-node-exporter-accelerators-collector-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/735df91f-0758-418c-8361-c7add0082060-node-exporter-textfile\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7gs\" (UniqueName: \"kubernetes.io/projected/735df91f-0758-418c-8361-c7add0082060-kube-api-access-nn7gs\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-root\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-node-exporter-wtmp\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.234572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.234418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.334987 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.334949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/735df91f-0758-418c-8361-c7add0082060-node-exporter-textfile\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.334987 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.334990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7gs\" (UniqueName: \"kubernetes.io/projected/735df91f-0758-418c-8361-c7add0082060-kube-api-access-nn7gs\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-root\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-node-exporter-wtmp\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-sys\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-metrics-client-ca\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-node-exporter-accelerators-collector-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-node-exporter-wtmp\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-root\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/735df91f-0758-418c-8361-c7add0082060-node-exporter-textfile\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:43.335393    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/735df91f-0758-418c-8361-c7add0082060-sys\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335584 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:18:43.335447    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls podName:735df91f-0758-418c-8361-c7add0082060 nodeName:}" failed. No retries permitted until 2026-09-11 21:18:43.835427592 +0000 UTC m=+71.957759497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls") pod "node-exporter-zr42f" (UID: "735df91f-0758-418c-8361-c7add0082060") : secret "node-exporter-tls" not found
Sep 11 21:18:43.335892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-node-exporter-accelerators-collector-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.335949 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.335911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735df91f-0758-418c-8361-c7add0082060-metrics-client-ca\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.338097 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.338073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.351290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.351266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7gs\" (UniqueName: \"kubernetes.io/projected/735df91f-0758-418c-8361-c7add0082060-kube-api-access-nn7gs\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.839721 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.839691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:43.842615 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:43.842592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/735df91f-0758-418c-8361-c7add0082060-node-exporter-tls\") pod \"node-exporter-zr42f\" (UID: \"735df91f-0758-418c-8361-c7add0082060\") " pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:44.024902 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:44.024871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zr42f"
Sep 11 21:18:44.035111 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:18:44.035084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735df91f_0758_418c_8361_c7add0082060.slice/crio-059abf78d94c87112e9ef12506e5cb79f8fa360634271aed7f827bfb6dbd9530 WatchSource:0}: Error finding container 059abf78d94c87112e9ef12506e5cb79f8fa360634271aed7f827bfb6dbd9530: Status 404 returned error can't find the container with id 059abf78d94c87112e9ef12506e5cb79f8fa360634271aed7f827bfb6dbd9530
Sep 11 21:18:44.739517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:44.739466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zr42f" event={"ID":"735df91f-0758-418c-8361-c7add0082060","Type":"ContainerStarted","Data":"059abf78d94c87112e9ef12506e5cb79f8fa360634271aed7f827bfb6dbd9530"}
Sep 11 21:18:44.741065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:44.741041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p7n7b" event={"ID":"75b7b403-2cec-4496-b4d5-11ac365f0305","Type":"ContainerStarted","Data":"12ed70ce906eb54a2a154b5beb49548d199b7dd20836781ef2bdec0c2183be49"}
Sep 11 21:18:44.767788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:44.767738    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p7n7b" podStartSLOduration=67.569989493 podStartE2EDuration="1m12.767721995s" totalImagesPullingTime="5.197732502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:44.76618327 +0000 UTC m=+72.888515203" watchObservedRunningTime="2026-09-11 21:18:44.767721995 +0000 UTC m=+72.890053916"
Sep 11 21:18:45.746426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:45.746395    2580 generic.go:356] "Generic (PLEG): container finished" podID="735df91f-0758-418c-8361-c7add0082060" containerID="32d673143067db7817f8a04e1fbffd5fe66b44da130bc22237e09bd515540682" exitCode=0
Sep 11 21:18:45.746813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:45.746484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zr42f" event={"ID":"735df91f-0758-418c-8361-c7add0082060","Type":"ContainerDied","Data":"32d673143067db7817f8a04e1fbffd5fe66b44da130bc22237e09bd515540682"}
Sep 11 21:18:46.752769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:46.752730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zr42f" event={"ID":"735df91f-0758-418c-8361-c7add0082060","Type":"ContainerStarted","Data":"b9bc8d52156a493d8336453b65995f61fe2ed468bc689cf6bea2cf41e406b94b"}
Sep 11 21:18:46.753224 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:46.752801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zr42f" event={"ID":"735df91f-0758-418c-8361-c7add0082060","Type":"ContainerStarted","Data":"877bd52efc80d1127dc916d6186062e2d1d3cd0d422ab179627c2e26cfd8b4a5"}
Sep 11 21:18:46.803813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:46.803766    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zr42f" podStartSLOduration=2.684977683 podStartE2EDuration="3.803750463s" totalImagesPullingTime="1.048280453s" totalInitContainerRuntime="70.492327ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:46.803364745 +0000 UTC m=+74.925696667" watchObservedRunningTime="2026-09-11 21:18:46.803750463 +0000 UTC m=+74.926082383"
Sep 11 21:18:54.692406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:18:54.692274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f578bb4fd-4dphg"
Sep 11 21:19:10.727824 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:19:10.727798    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-96jdf"
Sep 11 21:21:37.595658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.595621    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"]
Sep 11 21:21:37.597440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.597423    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.603740 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.603574    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:21:37.604561 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.604542    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k76nf\""
Sep 11 21:21:37.604657 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.604543    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:21:37.639211 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.639171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.639315 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.639263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvjh\" (UniqueName: \"kubernetes.io/projected/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-kube-api-access-kcvjh\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.641199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.641180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"]
Sep 11 21:21:37.739736 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.739703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvjh\" (UniqueName: \"kubernetes.io/projected/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-kube-api-access-kcvjh\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.739926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.739754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.740153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.740132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.752904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.752875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvjh\" (UniqueName: \"kubernetes.io/projected/e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065-kube-api-access-kcvjh\") pod \"openshift-lws-operator-fd8ccff4c-tgcgm\" (UID: \"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:37.906652 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:37.906570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"
Sep 11 21:21:38.207429 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:38.207354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm"]
Sep 11 21:21:38.210272 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:21:38.210242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f7f3c5_42e6_4523_96f6_ce5ee9a4a065.slice/crio-837202acdb75143636fe9a8c10bedc867b55bebb99be9c2b04c61a58f4675315 WatchSource:0}: Error finding container 837202acdb75143636fe9a8c10bedc867b55bebb99be9c2b04c61a58f4675315: Status 404 returned error can't find the container with id 837202acdb75143636fe9a8c10bedc867b55bebb99be9c2b04c61a58f4675315
Sep 11 21:21:38.321834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:38.321803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm" event={"ID":"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065","Type":"ContainerStarted","Data":"837202acdb75143636fe9a8c10bedc867b55bebb99be9c2b04c61a58f4675315"}
Sep 11 21:21:51.365853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:51.365814    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm" event={"ID":"e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065","Type":"ContainerStarted","Data":"71c45f27573a7881ac9d2bd66e649b475dd362c89dcef9b84af6094a9b7164a5"}
Sep 11 21:21:51.385582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:21:51.385539    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tgcgm" podStartSLOduration=2.240956727 podStartE2EDuration="14.385524495s" totalImagesPullingTime="12.144567768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:21:51.383710443 +0000 UTC m=+259.506042367" watchObservedRunningTime="2026-09-11 21:21:51.385524495 +0000 UTC m=+259.507856409"
Sep 11 21:22:09.677513 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.677424    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"]
Sep 11 21:22:09.679382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.679367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.682605 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.682585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:22:09.683212 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.683190    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:22:09.683352 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.683328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:22:09.683579 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.683562    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:22:09.683663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.683628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l4wkn\""
Sep 11 21:22:09.705129 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.705102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"]
Sep 11 21:22:09.772243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.772211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-manifests\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.772402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.772263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.772402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.772334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-tmp\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.772402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.772358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gc7j\" (UniqueName: \"kubernetes.io/projected/d0073dcb-15f2-4d2f-867d-02d542143d25-kube-api-access-5gc7j\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.772402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.772389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.872767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.872723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-tmp\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.872767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.872760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gc7j\" (UniqueName: \"kubernetes.io/projected/d0073dcb-15f2-4d2f-867d-02d542143d25-kube-api-access-5gc7j\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.873019 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.872796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.873019 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.872842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-manifests\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.873019 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.872887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.873297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.873252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-manifests\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.875836 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.875812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0073dcb-15f2-4d2f-867d-02d542143d25-tmp\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.876012 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.875996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.876154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.876132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d0073dcb-15f2-4d2f-867d-02d542143d25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.882618 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.882598    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gc7j\" (UniqueName: \"kubernetes.io/projected/d0073dcb-15f2-4d2f-867d-02d542143d25-kube-api-access-5gc7j\") pod \"opendatahub-operator-controller-manager-68b6c4779-mk7ff\" (UID: \"d0073dcb-15f2-4d2f-867d-02d542143d25\") " pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:09.988941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:09.988915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:10.266656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:10.266583    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"]
Sep 11 21:22:10.270670 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:10.270645    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0073dcb_15f2_4d2f_867d_02d542143d25.slice/crio-75c95ac0a8c1ff38b314760d8e1ae71aa99591d198c899fc0e78c290c550b112 WatchSource:0}: Error finding container 75c95ac0a8c1ff38b314760d8e1ae71aa99591d198c899fc0e78c290c550b112: Status 404 returned error can't find the container with id 75c95ac0a8c1ff38b314760d8e1ae71aa99591d198c899fc0e78c290c550b112
Sep 11 21:22:10.428000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:10.427966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff" event={"ID":"d0073dcb-15f2-4d2f-867d-02d542143d25","Type":"ContainerStarted","Data":"75c95ac0a8c1ff38b314760d8e1ae71aa99591d198c899fc0e78c290c550b112"}
Sep 11 21:22:13.442912 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:13.442878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff" event={"ID":"d0073dcb-15f2-4d2f-867d-02d542143d25","Type":"ContainerStarted","Data":"152acb6b2b8f9e10d448a99d5c9c1901733ca2b4b334dc83768ccf919bcea404"}
Sep 11 21:22:14.447757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:14.447726    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0073dcb-15f2-4d2f-867d-02d542143d25" containerID="152acb6b2b8f9e10d448a99d5c9c1901733ca2b4b334dc83768ccf919bcea404" exitCode=0
Sep 11 21:22:14.448106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:14.447763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff" event={"ID":"d0073dcb-15f2-4d2f-867d-02d542143d25","Type":"ContainerDied","Data":"152acb6b2b8f9e10d448a99d5c9c1901733ca2b4b334dc83768ccf919bcea404"}
Sep 11 21:22:15.454081 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:15.454047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff" event={"ID":"d0073dcb-15f2-4d2f-867d-02d542143d25","Type":"ContainerStarted","Data":"0485eee8571232a13cd0418adcc02e1a10014e8bd29d0ff48a889a849c9b7b25"}
Sep 11 21:22:15.454457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:15.454196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:15.481518 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:15.481449    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff" podStartSLOduration=2.138186603 podStartE2EDuration="6.481428537s" totalImagesPullingTime="3.264942253s" totalInitContainerRuntime="1.078299681s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:15.480327129 +0000 UTC m=+283.602659049" watchObservedRunningTime="2026-09-11 21:22:15.481428537 +0000 UTC m=+283.603760459"
Sep 11 21:22:26.460550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:26.460490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b6c4779-mk7ff"
Sep 11 21:22:32.306267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:32.306242    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:22:34.667127 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.667095    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"]
Sep 11 21:22:34.668935 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.668921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.671705 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.671675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 21:22:34.672042 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.672019    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:22:34.672180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.672129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 21:22:34.673310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.673294    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:22:34.704659 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.704633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"]
Sep 11 21:22:34.751028 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.751000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcftg\" (UniqueName: \"kubernetes.io/projected/dbfd34dc-f1d5-47cd-a824-7dcc62450056-kube-api-access-zcftg\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.751236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.751035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.751236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.751090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tmp\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.851747 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.851710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tmp\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.851886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.851778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcftg\" (UniqueName: \"kubernetes.io/projected/dbfd34dc-f1d5-47cd-a824-7dcc62450056-kube-api-access-zcftg\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.851886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.851805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.851958 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:34.851920    2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 21:22:34.852015 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:34.852004    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs podName:dbfd34dc-f1d5-47cd-a824-7dcc62450056 nodeName:}" failed. No retries permitted until 2026-09-11 21:22:35.351982191 +0000 UTC m=+303.474314092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs") pod "kube-auth-proxy-5564d7bb97-8d97v" (UID: "dbfd34dc-f1d5-47cd-a824-7dcc62450056") : secret "kube-auth-proxy-tls" not found
Sep 11 21:22:34.854097 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.854066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tmp\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:34.865562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:34.865539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcftg\" (UniqueName: \"kubernetes.io/projected/dbfd34dc-f1d5-47cd-a824-7dcc62450056-kube-api-access-zcftg\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:35.355439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:35.355401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:35.357954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:35.357929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd34dc-f1d5-47cd-a824-7dcc62450056-tls-certs\") pod \"kube-auth-proxy-5564d7bb97-8d97v\" (UID: \"dbfd34dc-f1d5-47cd-a824-7dcc62450056\") " pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:35.578770 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:35.578741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"
Sep 11 21:22:35.849834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:35.849802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v"]
Sep 11 21:22:35.852587 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:35.852562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd34dc_f1d5_47cd_a824_7dcc62450056.slice/crio-bbf4dfcd6bf48728f7eeaa97dbae74c058f993318f17c212930952c473c7b68f WatchSource:0}: Error finding container bbf4dfcd6bf48728f7eeaa97dbae74c058f993318f17c212930952c473c7b68f: Status 404 returned error can't find the container with id bbf4dfcd6bf48728f7eeaa97dbae74c058f993318f17c212930952c473c7b68f
Sep 11 21:22:35.854461 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:35.854441    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:22:36.029274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.029244    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:36.031733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.031712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.034479 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.034460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 21:22:36.034734 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.034718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2765v\""
Sep 11 21:22:36.034979 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.034959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 21:22:36.035023 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.035009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 21:22:36.035076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.035009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:22:36.051076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.051046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:36.161730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hb2h\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.161995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.161916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263325 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263451 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263495 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263622 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hb2h\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.263873 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.263827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.264067 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.264036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.266012 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.265992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.266102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.266023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.266102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.266027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.266173 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.266117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.272617 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.272592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.273034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.273010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hb2h\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h\") pod \"istiod-openshift-gateway-7b7995d5cd-hgjd9\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.340804 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.340731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:36.530168 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.530102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v" event={"ID":"dbfd34dc-f1d5-47cd-a824-7dcc62450056","Type":"ContainerStarted","Data":"bbf4dfcd6bf48728f7eeaa97dbae74c058f993318f17c212930952c473c7b68f"}
Sep 11 21:22:36.617313 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:36.617037    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:36.619726 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:36.619694    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b76c1f_73c9_4f55_aee4_1afd075a255d.slice/crio-c73fdef376e5e2fbe019b63302ab618cbccb00430604bf933935d343d5d2df18 WatchSource:0}: Error finding container c73fdef376e5e2fbe019b63302ab618cbccb00430604bf933935d343d5d2df18: Status 404 returned error can't find the container with id c73fdef376e5e2fbe019b63302ab618cbccb00430604bf933935d343d5d2df18
Sep 11 21:22:37.535639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:37.535595    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" event={"ID":"15b76c1f-73c9-4f55-aee4-1afd075a255d","Type":"ContainerStarted","Data":"c73fdef376e5e2fbe019b63302ab618cbccb00430604bf933935d343d5d2df18"}
Sep 11 21:22:39.546392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:39.546353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v" event={"ID":"dbfd34dc-f1d5-47cd-a824-7dcc62450056","Type":"ContainerStarted","Data":"85db90d4a78a9b9c26373e17530b777dd51e3c169debc3e43599e2359587a602"}
Sep 11 21:22:39.565596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:39.565457    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5564d7bb97-8d97v" podStartSLOduration=2.898238371 podStartE2EDuration="5.565439466s" totalImagesPullingTime="2.667201095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:39.56373435 +0000 UTC m=+307.686066272" watchObservedRunningTime="2026-09-11 21:22:39.565439466 +0000 UTC m=+307.687771388"
Sep 11 21:22:40.267675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:40.267621    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:22:40.267757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:40.267717    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:22:40.553239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:40.553134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" event={"ID":"15b76c1f-73c9-4f55-aee4-1afd075a255d","Type":"ContainerStarted","Data":"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"}
Sep 11 21:22:40.575379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:40.575329    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" podStartSLOduration=0.93080015 podStartE2EDuration="4.575313374s" totalImagesPullingTime="3.644513224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:40.574265566 +0000 UTC m=+308.696597489" watchObservedRunningTime="2026-09-11 21:22:40.575313374 +0000 UTC m=+308.697645293"
Sep 11 21:22:41.558695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:41.558663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:41.564819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:41.564791    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-hgjd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:22:41.565048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:41.565020    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:22:42.266381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.266346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"]
Sep 11 21:22:42.269021 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.269000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.271626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.271605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-79rf5\""
Sep 11 21:22:42.283391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.283365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"]
Sep 11 21:22:42.319077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktnq\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-kube-api-access-wktnq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.319297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.319265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbe602-3995-40a8-aa9e-d17dfa72d340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.419922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.419888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.419922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.419921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.419940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbe602-3995-40a8-aa9e-d17dfa72d340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.419966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.419997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktnq\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-kube-api-access-wktnq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.420890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.420807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.421072 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.421049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28dbe602-3995-40a8-aa9e-d17dfa72d340-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.422442 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.422417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.422613 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.422599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.428496 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.428467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktnq\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-kube-api-access-wktnq\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.428635 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.428571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28dbe602-3995-40a8-aa9e-d17dfa72d340-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2\" (UID: \"28dbe602-3995-40a8-aa9e-d17dfa72d340\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.563913 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.563839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:42.579851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.579818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:42.862044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:42.861945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"]
Sep 11 21:22:42.865742 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:42.865712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dbe602_3995_40a8_aa9e_d17dfa72d340.slice/crio-a791d2e661b7ed4a80a4cf02f06b9cf3724067baa7779314c9369bb143cdcc0d WatchSource:0}: Error finding container a791d2e661b7ed4a80a4cf02f06b9cf3724067baa7779314c9369bb143cdcc0d: Status 404 returned error can't find the container with id a791d2e661b7ed4a80a4cf02f06b9cf3724067baa7779314c9369bb143cdcc0d
Sep 11 21:22:43.568687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:43.568636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2" event={"ID":"28dbe602-3995-40a8-aa9e-d17dfa72d340","Type":"ContainerStarted","Data":"a791d2e661b7ed4a80a4cf02f06b9cf3724067baa7779314c9369bb143cdcc0d"}
Sep 11 21:22:45.385495 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.385458    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:22:45.385762 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.385543    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:22:45.385762 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.385575    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:22:45.579396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.579318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2" event={"ID":"28dbe602-3995-40a8-aa9e-d17dfa72d340","Type":"ContainerStarted","Data":"66d6c6a431c71959508dce8b2a333c8983f2ca2a4a93d1fdb57c9f42bde8192e"}
Sep 11 21:22:45.580278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.580264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:45.602086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:45.602039    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2" podStartSLOduration=1.085374349 podStartE2EDuration="3.602022907s" totalImagesPullingTime="2.516648558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:45.601281568 +0000 UTC m=+313.723613491" watchObservedRunningTime="2026-09-11 21:22:45.602022907 +0000 UTC m=+313.724354889"
Sep 11 21:22:46.585741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:46.585714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:47.588580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:47.588549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:47.589628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:47.589609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2"
Sep 11 21:22:52.028792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.028761    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klktt"]
Sep 11 21:22:52.031877 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.031860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.034375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.034349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 21:22:52.034471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.034407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rcjfq\""
Sep 11 21:22:52.045783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.045759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klktt"]
Sep 11 21:22:52.056376 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.056350    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"]
Sep 11 21:22:52.059692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.059670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.062645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.062626    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 21:22:52.064031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.063865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mc4vt\""
Sep 11 21:22:52.075359 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.075337    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"]
Sep 11 21:22:52.098022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.096589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.098022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.096651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.098022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.096685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f975z\" (UniqueName: \"kubernetes.io/projected/e1e990a0-3ad7-4963-912f-5263171aed36-kube-api-access-f975z\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.098022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.096767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8hj\" (UniqueName: \"kubernetes.io/projected/6cad4815-9575-4f2d-bc03-def80dbb4f2d-kube-api-access-tx8hj\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.197693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.197659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f975z\" (UniqueName: \"kubernetes.io/projected/e1e990a0-3ad7-4963-912f-5263171aed36-kube-api-access-f975z\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.197889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.197720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8hj\" (UniqueName: \"kubernetes.io/projected/6cad4815-9575-4f2d-bc03-def80dbb4f2d-kube-api-access-tx8hj\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.197889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.197754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.197889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.197792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.197889 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:52.197888    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 21:22:52.198072 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:52.197913    2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 21:22:52.198072 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:52.197973    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert podName:6cad4815-9575-4f2d-bc03-def80dbb4f2d nodeName:}" failed. No retries permitted until 2026-09-11 21:22:52.697952679 +0000 UTC m=+320.820284592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert") pod "kserve-controller-manager-d6f4b65b8-klktt" (UID: "6cad4815-9575-4f2d-bc03-def80dbb4f2d") : secret "kserve-webhook-server-cert" not found
Sep 11 21:22:52.198072 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:52.197989    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert podName:e1e990a0-3ad7-4963-912f-5263171aed36 nodeName:}" failed. No retries permitted until 2026-09-11 21:22:52.697981743 +0000 UTC m=+320.820313643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert") pod "llmisvc-controller-manager-777c6bd7d5-m55xd" (UID: "e1e990a0-3ad7-4963-912f-5263171aed36") : secret "llmisvc-webhook-server-cert" not found
Sep 11 21:22:52.217407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.217369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8hj\" (UniqueName: \"kubernetes.io/projected/6cad4815-9575-4f2d-bc03-def80dbb4f2d-kube-api-access-tx8hj\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.219871 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.219843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f975z\" (UniqueName: \"kubernetes.io/projected/e1e990a0-3ad7-4963-912f-5263171aed36-kube-api-access-f975z\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.701695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.701662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.701695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.701703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.704239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.704210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1e990a0-3ad7-4963-912f-5263171aed36-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m55xd\" (UID: \"e1e990a0-3ad7-4963-912f-5263171aed36\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:52.704335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.704247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cad4815-9575-4f2d-bc03-def80dbb4f2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-klktt\" (UID: \"6cad4815-9575-4f2d-bc03-def80dbb4f2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.944933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.944902    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:52.973851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:52.973825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:53.221808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.217445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klktt"]
Sep 11 21:22:53.223546 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:53.223489    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cad4815_9575_4f2d_bc03_def80dbb4f2d.slice/crio-66a2cc306667178cf6533dae048e00a9fa2b432856d675367932d64731538a63 WatchSource:0}: Error finding container 66a2cc306667178cf6533dae048e00a9fa2b432856d675367932d64731538a63: Status 404 returned error can't find the container with id 66a2cc306667178cf6533dae048e00a9fa2b432856d675367932d64731538a63
Sep 11 21:22:53.261195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.261173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"]
Sep 11 21:22:53.263419 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:22:53.263387    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1e990a0_3ad7_4963_912f_5263171aed36.slice/crio-fca41cf92979fdfcec1db476dcd32fcbdec8f5eaa924b4ba15c84c1ecf2fbccb WatchSource:0}: Error finding container fca41cf92979fdfcec1db476dcd32fcbdec8f5eaa924b4ba15c84c1ecf2fbccb: Status 404 returned error can't find the container with id fca41cf92979fdfcec1db476dcd32fcbdec8f5eaa924b4ba15c84c1ecf2fbccb
Sep 11 21:22:53.513895 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.513870    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:53.514060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.514040    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerName="discovery" containerID="cri-o://59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a" gracePeriod=30
Sep 11 21:22:53.614083 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.614047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt" event={"ID":"6cad4815-9575-4f2d-bc03-def80dbb4f2d","Type":"ContainerStarted","Data":"66a2cc306667178cf6533dae048e00a9fa2b432856d675367932d64731538a63"}
Sep 11 21:22:53.615184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.615165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd" event={"ID":"e1e990a0-3ad7-4963-912f-5263171aed36","Type":"ContainerStarted","Data":"fca41cf92979fdfcec1db476dcd32fcbdec8f5eaa924b4ba15c84c1ecf2fbccb"}
Sep 11 21:22:53.783950 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.783923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:53.809479 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809517    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809652    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hb2h\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.809838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.809677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert\") pod \"15b76c1f-73c9-4f55-aee4-1afd075a255d\" (UID: \"15b76c1f-73c9-4f55-aee4-1afd075a255d\") "
Sep 11 21:22:53.810742 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.810687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:22:53.813744 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.813710    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:22:53.813845 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.813823    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts" (OuterVolumeSpecName: "cacerts") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:22:53.814349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.813996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:22:53.814349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.814197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h" (OuterVolumeSpecName: "kube-api-access-6hb2h") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "kube-api-access-6hb2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:53.814537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.814514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs" (OuterVolumeSpecName: "local-certs") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:53.815807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.815783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token" (OuterVolumeSpecName: "istio-token") pod "15b76c1f-73c9-4f55-aee4-1afd075a255d" (UID: "15b76c1f-73c9-4f55-aee4-1afd075a255d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:53.910310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910282    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-kubeconfig\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910306    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b76c1f-73c9-4f55-aee4-1afd075a255d-local-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910317    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hb2h\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-kube-api-access-6hb2h\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910326    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-dns-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910334    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/15b76c1f-73c9-4f55-aee4-1afd075a255d-cacerts\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910344    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:53.910549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:53.910352    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/15b76c1f-73c9-4f55-aee4-1afd075a255d-istio-csr-ca-configmap\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:22:54.624589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.624555    2580 generic.go:356] "Generic (PLEG): container finished" podID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerID="59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a" exitCode=0
Sep 11 21:22:54.625062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.624615    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"
Sep 11 21:22:54.625062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.624635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" event={"ID":"15b76c1f-73c9-4f55-aee4-1afd075a255d","Type":"ContainerDied","Data":"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"}
Sep 11 21:22:54.625062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.624684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9" event={"ID":"15b76c1f-73c9-4f55-aee4-1afd075a255d","Type":"ContainerDied","Data":"c73fdef376e5e2fbe019b63302ab618cbccb00430604bf933935d343d5d2df18"}
Sep 11 21:22:54.625062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.624712    2580 scope.go:117] "RemoveContainer" containerID="59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"
Sep 11 21:22:54.636807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.636782    2580 scope.go:117] "RemoveContainer" containerID="59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"
Sep 11 21:22:54.637127 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:22:54.637099    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a\": container with ID starting with 59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a not found: ID does not exist" containerID="59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"
Sep 11 21:22:54.637195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.637134    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a"} err="failed to get container status \"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a\": rpc error: code = NotFound desc = could not find container \"59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a\": container with ID starting with 59948e595304b57c7b993c21a3402be3c640c04e9d808b927f93a4c7d8b6449a not found: ID does not exist"
Sep 11 21:22:54.649582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.649469    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:54.659719 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:54.659691    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hgjd9"]
Sep 11 21:22:55.631611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:55.631578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt" event={"ID":"6cad4815-9575-4f2d-bc03-def80dbb4f2d","Type":"ContainerStarted","Data":"bd61cfa8217af35bdbb999343d5a837dd1e20ab358cfd5d038515f7eff6ea04c"}
Sep 11 21:22:55.632096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:55.631689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:22:55.692187 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:55.692082    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt" podStartSLOduration=1.5101758730000001 podStartE2EDuration="3.692065673s" totalImagesPullingTime="2.1818898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:55.690314746 +0000 UTC m=+323.812646669" watchObservedRunningTime="2026-09-11 21:22:55.692065673 +0000 UTC m=+323.814397593"
Sep 11 21:22:56.382421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:56.382344    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" path="/var/lib/kubelet/pods/15b76c1f-73c9-4f55-aee4-1afd075a255d/volumes"
Sep 11 21:22:56.638075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:56.637983    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd" event={"ID":"e1e990a0-3ad7-4963-912f-5263171aed36","Type":"ContainerStarted","Data":"aa61c7d366e20cffdc57e038840cb321698ee0fd9385070e7a506eafab1348d9"}
Sep 11 21:22:56.638433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:56.638124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:22:56.659779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:22:56.659733    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd" podStartSLOduration=1.84617986 podStartE2EDuration="4.659719051s" totalImagesPullingTime="2.813539191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:56.658666315 +0000 UTC m=+324.780998271" watchObservedRunningTime="2026-09-11 21:22:56.659719051 +0000 UTC m=+324.782050975"
Sep 11 21:23:26.643401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:26.643309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klktt"
Sep 11 21:23:27.645485 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:27.645457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m55xd"
Sep 11 21:23:58.963019 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.962988    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-k7mtz"]
Sep 11 21:23:58.963478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.963278    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerName="discovery"
Sep 11 21:23:58.963478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.963290    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerName="discovery"
Sep 11 21:23:58.963478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.963351    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15b76c1f-73c9-4f55-aee4-1afd075a255d" containerName="discovery"
Sep 11 21:23:58.966218 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.966203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:23:58.969418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.969396    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qjzmf\""
Sep 11 21:23:58.969630 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.969616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:23:58.970314 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.970301    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:23:58.979964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:58.979938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-k7mtz"]
Sep 11 21:23:59.134545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.134484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97td\" (UniqueName: \"kubernetes.io/projected/33ae7105-b658-41b8-aec1-2c508ff15f2c-kube-api-access-v97td\") pod \"authorino-operator-76b478bd78-k7mtz\" (UID: \"33ae7105-b658-41b8-aec1-2c508ff15f2c\") " pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:23:59.235110 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.235005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97td\" (UniqueName: \"kubernetes.io/projected/33ae7105-b658-41b8-aec1-2c508ff15f2c-kube-api-access-v97td\") pod \"authorino-operator-76b478bd78-k7mtz\" (UID: \"33ae7105-b658-41b8-aec1-2c508ff15f2c\") " pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:23:59.249727 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.249697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97td\" (UniqueName: \"kubernetes.io/projected/33ae7105-b658-41b8-aec1-2c508ff15f2c-kube-api-access-v97td\") pod \"authorino-operator-76b478bd78-k7mtz\" (UID: \"33ae7105-b658-41b8-aec1-2c508ff15f2c\") " pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:23:59.276484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.276457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:23:59.557117 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.557089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-k7mtz"]
Sep 11 21:23:59.559873 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:23:59.559846    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ae7105_b658_41b8_aec1_2c508ff15f2c.slice/crio-cff81075b28823ec01519c9f35fd05b9c7773d91019e6dc2abd46fe2135feb1f WatchSource:0}: Error finding container cff81075b28823ec01519c9f35fd05b9c7773d91019e6dc2abd46fe2135feb1f: Status 404 returned error can't find the container with id cff81075b28823ec01519c9f35fd05b9c7773d91019e6dc2abd46fe2135feb1f
Sep 11 21:23:59.889104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:23:59.889018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz" event={"ID":"33ae7105-b658-41b8-aec1-2c508ff15f2c","Type":"ContainerStarted","Data":"cff81075b28823ec01519c9f35fd05b9c7773d91019e6dc2abd46fe2135feb1f"}
Sep 11 21:24:01.141577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.141547    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"]
Sep 11 21:24:01.150373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.150342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:01.153658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.153633    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qq9z8\""
Sep 11 21:24:01.153788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.153698    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 21:24:01.173965 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.173930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"]
Sep 11 21:24:01.251731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.251698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mjd\" (UniqueName: \"kubernetes.io/projected/d4394a32-ee6d-43f5-a240-f9d9d92d145c-kube-api-access-r5mjd\") pod \"dns-operator-controller-manager-7b84cd7c9c-nqjqh\" (UID: \"d4394a32-ee6d-43f5-a240-f9d9d92d145c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:01.353172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.353134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mjd\" (UniqueName: \"kubernetes.io/projected/d4394a32-ee6d-43f5-a240-f9d9d92d145c-kube-api-access-r5mjd\") pod \"dns-operator-controller-manager-7b84cd7c9c-nqjqh\" (UID: \"d4394a32-ee6d-43f5-a240-f9d9d92d145c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:01.386544 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.386495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mjd\" (UniqueName: \"kubernetes.io/projected/d4394a32-ee6d-43f5-a240-f9d9d92d145c-kube-api-access-r5mjd\") pod \"dns-operator-controller-manager-7b84cd7c9c-nqjqh\" (UID: \"d4394a32-ee6d-43f5-a240-f9d9d92d145c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:01.462798 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:01.462714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:02.082927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:02.082833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"]
Sep 11 21:24:02.086804 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:02.086771    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4394a32_ee6d_43f5_a240_f9d9d92d145c.slice/crio-f59343764e1f3bdf8db0d13a0bf8f54bdb7abf61a5b8687811881295d0411cf9 WatchSource:0}: Error finding container f59343764e1f3bdf8db0d13a0bf8f54bdb7abf61a5b8687811881295d0411cf9: Status 404 returned error can't find the container with id f59343764e1f3bdf8db0d13a0bf8f54bdb7abf61a5b8687811881295d0411cf9
Sep 11 21:24:02.905033 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:02.905002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz" event={"ID":"33ae7105-b658-41b8-aec1-2c508ff15f2c","Type":"ContainerStarted","Data":"5db727eeb4bfc3e9d3fe41fc7ae1d160eba81c2b91341973ee699a2b4493451c"}
Sep 11 21:24:02.905473 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:02.905121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:24:02.906418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:02.906396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh" event={"ID":"d4394a32-ee6d-43f5-a240-f9d9d92d145c","Type":"ContainerStarted","Data":"f59343764e1f3bdf8db0d13a0bf8f54bdb7abf61a5b8687811881295d0411cf9"}
Sep 11 21:24:02.927754 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:02.927705    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz" podStartSLOduration=2.613177935 podStartE2EDuration="4.927689444s" totalImagesPullingTime="2.314511509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:02.926629486 +0000 UTC m=+391.048961433" watchObservedRunningTime="2026-09-11 21:24:02.927689444 +0000 UTC m=+391.050021365"
Sep 11 21:24:04.915427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:04.915390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh" event={"ID":"d4394a32-ee6d-43f5-a240-f9d9d92d145c","Type":"ContainerStarted","Data":"8b6cc053330c957d07e539f4076b6a800f5e31a231486ca64006583d4e946206"}
Sep 11 21:24:04.915831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:04.915535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:04.941766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:04.941718    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh" podStartSLOduration=2.12974698 podStartE2EDuration="3.941701867s" totalImagesPullingTime="1.811954887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:04.939134698 +0000 UTC m=+393.061466618" watchObservedRunningTime="2026-09-11 21:24:04.941701867 +0000 UTC m=+393.064033877"
Sep 11 21:24:05.596543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.596496    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"]
Sep 11 21:24:05.599625 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.599609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.602655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.602625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-chzk9\""
Sep 11 21:24:05.617326 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.617305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"]
Sep 11 21:24:05.690515 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.690461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.690677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.690583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmrx\" (UniqueName: \"kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.791629 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.791578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmrx\" (UniqueName: \"kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.791803 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.791647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.792023 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.792005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.802820 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.802796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmrx\" (UniqueName: \"kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:05.909298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:05.909213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:06.200709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:06.200627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"]
Sep 11 21:24:06.204525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:06.204484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188e069f_2b20_4d23_b5bc_d39f70227358.slice/crio-b6bf907cc99b3b4868f2ffcec64ea3f533e1b8e13cde5da52c9bf16ae5a4b790 WatchSource:0}: Error finding container b6bf907cc99b3b4868f2ffcec64ea3f533e1b8e13cde5da52c9bf16ae5a4b790: Status 404 returned error can't find the container with id b6bf907cc99b3b4868f2ffcec64ea3f533e1b8e13cde5da52c9bf16ae5a4b790
Sep 11 21:24:06.925695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:06.925661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg" event={"ID":"188e069f-2b20-4d23-b5bc-d39f70227358","Type":"ContainerStarted","Data":"b6bf907cc99b3b4868f2ffcec64ea3f533e1b8e13cde5da52c9bf16ae5a4b790"}
Sep 11 21:24:07.483055 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.483017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"]
Sep 11 21:24:07.486441 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.486416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:07.489426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.489402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jbqmn\""
Sep 11 21:24:07.498115 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.498095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"]
Sep 11 21:24:07.507954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.507931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8wd\" (UniqueName: \"kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd\") pod \"limitador-operator-controller-manager-74495c69f8-bphzp\" (UID: \"2ac09c1d-4334-428f-b92f-a26d59e0899d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:07.608320 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.608283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8wd\" (UniqueName: \"kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd\") pod \"limitador-operator-controller-manager-74495c69f8-bphzp\" (UID: \"2ac09c1d-4334-428f-b92f-a26d59e0899d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:07.621625 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.621595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8wd\" (UniqueName: \"kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd\") pod \"limitador-operator-controller-manager-74495c69f8-bphzp\" (UID: \"2ac09c1d-4334-428f-b92f-a26d59e0899d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:07.799006 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:07.798973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:08.090433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:08.090353    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"]
Sep 11 21:24:08.093585 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:08.093559    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac09c1d_4334_428f_b92f_a26d59e0899d.slice/crio-1e78d2a6c8e45954e7e1e7d3bd31e3e91c9752da2b90319aac07c004148d2574 WatchSource:0}: Error finding container 1e78d2a6c8e45954e7e1e7d3bd31e3e91c9752da2b90319aac07c004148d2574: Status 404 returned error can't find the container with id 1e78d2a6c8e45954e7e1e7d3bd31e3e91c9752da2b90319aac07c004148d2574
Sep 11 21:24:08.941797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:08.941756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp" event={"ID":"2ac09c1d-4334-428f-b92f-a26d59e0899d","Type":"ContainerStarted","Data":"1e78d2a6c8e45954e7e1e7d3bd31e3e91c9752da2b90319aac07c004148d2574"}
Sep 11 21:24:10.953286 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:10.953204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg" event={"ID":"188e069f-2b20-4d23-b5bc-d39f70227358","Type":"ContainerStarted","Data":"015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78"}
Sep 11 21:24:10.953702 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:10.953323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:10.955022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:10.955003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp" event={"ID":"2ac09c1d-4334-428f-b92f-a26d59e0899d","Type":"ContainerStarted","Data":"ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f"}
Sep 11 21:24:10.955158 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:10.955146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:10.975660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:10.975621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg" podStartSLOduration=1.569450499 podStartE2EDuration="5.975609001s" totalImagesPullingTime="4.406158502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:10.975329665 +0000 UTC m=+399.097661586" watchObservedRunningTime="2026-09-11 21:24:10.975609001 +0000 UTC m=+399.097940969"
Sep 11 21:24:11.000095 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:11.000052    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp" podStartSLOduration=1.49047627 podStartE2EDuration="4.000039278s" totalImagesPullingTime="2.509563008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:10.9983877 +0000 UTC m=+399.120719634" watchObservedRunningTime="2026-09-11 21:24:11.000039278 +0000 UTC m=+399.122371224"
Sep 11 21:24:13.913282 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:13.913246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-k7mtz"
Sep 11 21:24:15.922389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:15.922358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-nqjqh"
Sep 11 21:24:21.963149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:21.963106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:21.963532 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:21.963185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.423973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.423943    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"]
Sep 11 21:24:24.424388 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.424125    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" containerName="manager" containerID="cri-o://015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78" gracePeriod=2
Sep 11 21:24:24.449363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.449337    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"]
Sep 11 21:24:24.473419 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.473379    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"188e069f-2b20-4d23-b5bc-d39f70227358\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\": pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.484223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.484191    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"]
Sep 11 21:24:24.484522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.484475    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" containerName="manager" containerID="cri-o://ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f" gracePeriod=2
Sep 11 21:24:24.497077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.497053    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"]
Sep 11 21:24:24.522784 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.522751    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2ac09c1d-4334-428f-b92f-a26d59e0899d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-bphzp\": pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.524114 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524090    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:24.524366 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524353    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" containerName="manager"
Sep 11 21:24:24.524366 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524365    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" containerName="manager"
Sep 11 21:24:24.524517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524378    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" containerName="manager"
Sep 11 21:24:24.524517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524383    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" containerName="manager"
Sep 11 21:24:24.524517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524437    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" containerName="manager"
Sep 11 21:24:24.524517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.524444    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" containerName="manager"
Sep 11 21:24:24.525643 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.525615    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.527289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.527272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.536059 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.536033    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.542331 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.542181    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.544992 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.544970    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:24.555986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.555961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"]
Sep 11 21:24:24.559065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.559041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:24.576070 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.576048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"]
Sep 11 21:24:24.578688 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.578661    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.580701 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.580672    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.653101 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.653046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69n2q\" (UniqueName: \"kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.653257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.653123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjzk\" (UniqueName: \"kubernetes.io/projected/f66f6191-d59c-447a-9b4c-dffba63c19aa-kube-api-access-hdjzk\") pod \"limitador-operator-controller-manager-74495c69f8-dpfwr\" (UID: \"f66f6191-d59c-447a-9b4c-dffba63c19aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:24.653257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.653159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.724268 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.724244    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.726469 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.726443    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.726791 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.726774    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.728485 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.728462    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.730440 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.730416    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:24.732361 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:24.732338    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:24.753752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.753730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69n2q\" (UniqueName: \"kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.753819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.753776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjzk\" (UniqueName: \"kubernetes.io/projected/f66f6191-d59c-447a-9b4c-dffba63c19aa-kube-api-access-hdjzk\") pod \"limitador-operator-controller-manager-74495c69f8-dpfwr\" (UID: \"f66f6191-d59c-447a-9b4c-dffba63c19aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:24.753819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.753800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.754146 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.754130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.772861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.772835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjzk\" (UniqueName: \"kubernetes.io/projected/f66f6191-d59c-447a-9b4c-dffba63c19aa-kube-api-access-hdjzk\") pod \"limitador-operator-controller-manager-74495c69f8-dpfwr\" (UID: \"f66f6191-d59c-447a-9b4c-dffba63c19aa\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:24.772941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.772906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69n2q\" (UniqueName: \"kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mgzdx\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.854895 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.854871    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx8wd\" (UniqueName: \"kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd\") pod \"2ac09c1d-4334-428f-b92f-a26d59e0899d\" (UID: \"2ac09c1d-4334-428f-b92f-a26d59e0899d\") "
Sep 11 21:24:24.855007 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.854951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmrx\" (UniqueName: \"kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx\") pod \"188e069f-2b20-4d23-b5bc-d39f70227358\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") "
Sep 11 21:24:24.855007 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.854978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume\") pod \"188e069f-2b20-4d23-b5bc-d39f70227358\" (UID: \"188e069f-2b20-4d23-b5bc-d39f70227358\") "
Sep 11 21:24:24.855588 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.855479    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "188e069f-2b20-4d23-b5bc-d39f70227358" (UID: "188e069f-2b20-4d23-b5bc-d39f70227358"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:24:24.856421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.856399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:24.857039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.857016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd" (OuterVolumeSpecName: "kube-api-access-sx8wd") pod "2ac09c1d-4334-428f-b92f-a26d59e0899d" (UID: "2ac09c1d-4334-428f-b92f-a26d59e0899d"). InnerVolumeSpecName "kube-api-access-sx8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:24:24.857142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.857120    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx" (OuterVolumeSpecName: "kube-api-access-scmrx") pod "188e069f-2b20-4d23-b5bc-d39f70227358" (UID: "188e069f-2b20-4d23-b5bc-d39f70227358"). InnerVolumeSpecName "kube-api-access-scmrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:24:24.868572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.868551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:24.955853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.955789    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmrx\" (UniqueName: \"kubernetes.io/projected/188e069f-2b20-4d23-b5bc-d39f70227358-kube-api-access-scmrx\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:24:24.955853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.955814    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/188e069f-2b20-4d23-b5bc-d39f70227358-extensions-socket-volume\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:24:24.955853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:24.955825    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8wd\" (UniqueName: \"kubernetes.io/projected/2ac09c1d-4334-428f-b92f-a26d59e0899d-kube-api-access-sx8wd\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:24:25.017275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.017244    2580 generic.go:356] "Generic (PLEG): container finished" podID="188e069f-2b20-4d23-b5bc-d39f70227358" containerID="015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78" exitCode=0
Sep 11 21:24:25.017432 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.017294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.017432 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.017350    2580 scope.go:117] "RemoveContainer" containerID="015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78"
Sep 11 21:24:25.019062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.019025    2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac09c1d-4334-428f-b92f-a26d59e0899d" containerID="ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f" exitCode=0
Sep 11 21:24:25.019154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.019079    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.019937 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.019911    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.021789 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.021762    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.024550 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.024521    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.025636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.025615    2580 scope.go:117] "RemoveContainer" containerID="015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78"
Sep 11 21:24:25.025945 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.025922    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78\": container with ID starting with 015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78 not found: ID does not exist" containerID="015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78"
Sep 11 21:24:25.026015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.025959    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78"} err="failed to get container status \"015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78\": rpc error: code = NotFound desc = could not find container \"015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78\": container with ID starting with 015031d45118cf746aba5355104a5a4237d595d6e7016f62ea440814386a6a78 not found: ID does not exist"
Sep 11 21:24:25.026015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.025982    2580 scope.go:117] "RemoveContainer" containerID="ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f"
Sep 11 21:24:25.026662 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.026639    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.032058 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.032021    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.033981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.033963    2580 scope.go:117] "RemoveContainer" containerID="ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f"
Sep 11 21:24:25.034187 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.034158    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.034289 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.034269    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f\": container with ID starting with ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f not found: ID does not exist" containerID="ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f"
Sep 11 21:24:25.034330 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.034298    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f"} err="failed to get container status \"ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f\": rpc error: code = NotFound desc = could not find container \"ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f\": container with ID starting with ad930dcff22098d26b4ad1dbf612049bcdf6ca56966038d0b5bc9df7d3581c1f not found: ID does not exist"
Sep 11 21:24:25.036469 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.036445    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.043017 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.042996    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.143299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.143276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:25.145536 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:25.145487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b19bf9_b070_42e5_af11_587f8d3b6290.slice/crio-894ecef96f655cf3dc9b8c291c7ca6c90579a0d49835181c1291198ec8248038 WatchSource:0}: Error finding container 894ecef96f655cf3dc9b8c291c7ca6c90579a0d49835181c1291198ec8248038: Status 404 returned error can't find the container with id 894ecef96f655cf3dc9b8c291c7ca6c90579a0d49835181c1291198ec8248038
Sep 11 21:24:25.153002 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.152983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"]
Sep 11 21:24:25.154950 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:25.154926    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66f6191_d59c_447a_9b4c_dffba63c19aa.slice/crio-b7ec06d269e4a5da862e2a5b4be0485aebff36439a8f6b544ac3b7a36217cdaa WatchSource:0}: Error finding container b7ec06d269e4a5da862e2a5b4be0485aebff36439a8f6b544ac3b7a36217cdaa: Status 404 returned error can't find the container with id b7ec06d269e4a5da862e2a5b4be0485aebff36439a8f6b544ac3b7a36217cdaa
Sep 11 21:24:25.409550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.409517    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:24:25.412794 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.412779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.414969 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.414941    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:25.417382 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:25.417349    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:25.447100 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.447076    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:24:25.459288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.459265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.459389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.459368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8dn\" (UniqueName: \"kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.559819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.559784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8dn\" (UniqueName: \"kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.559960 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.559829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.560142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.560127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.570781 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.570762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8dn\" (UniqueName: \"kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d9dls\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:25.722725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:25.722639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:26.004104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.004073    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:24:26.007207 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:26.007183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc842d0a_b4ea_4cda_9fa9_b01d8d18cd0f.slice/crio-256b010893247662c64f6dc15ccc805ba343eff98a2f2af661aa81aaed09194b WatchSource:0}: Error finding container 256b010893247662c64f6dc15ccc805ba343eff98a2f2af661aa81aaed09194b: Status 404 returned error can't find the container with id 256b010893247662c64f6dc15ccc805ba343eff98a2f2af661aa81aaed09194b
Sep 11 21:24:26.026974 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.026942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" event={"ID":"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f","Type":"ContainerStarted","Data":"256b010893247662c64f6dc15ccc805ba343eff98a2f2af661aa81aaed09194b"}
Sep 11 21:24:26.031713 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.031681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" event={"ID":"27b19bf9-b070-42e5-af11-587f8d3b6290","Type":"ContainerStarted","Data":"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"}
Sep 11 21:24:26.031813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.031741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" event={"ID":"27b19bf9-b070-42e5-af11-587f8d3b6290","Type":"ContainerStarted","Data":"894ecef96f655cf3dc9b8c291c7ca6c90579a0d49835181c1291198ec8248038"}
Sep 11 21:24:26.032102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.032050    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:26.033921 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.033896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr" event={"ID":"f66f6191-d59c-447a-9b4c-dffba63c19aa","Type":"ContainerStarted","Data":"89f0a4535841c137daac998dd61fe49a3b3193f78d4c916ffb1e1e5de52b37eb"}
Sep 11 21:24:26.034000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.033943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr" event={"ID":"f66f6191-d59c-447a-9b4c-dffba63c19aa","Type":"ContainerStarted","Data":"b7ec06d269e4a5da862e2a5b4be0485aebff36439a8f6b544ac3b7a36217cdaa"}
Sep 11 21:24:26.034000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.033979    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:26.034336 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:26.034311    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:26.036192 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:26.036168    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:26.056450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.056409    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" podStartSLOduration=2.05639425 podStartE2EDuration="2.05639425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:26.05541349 +0000 UTC m=+414.177745414" watchObservedRunningTime="2026-09-11 21:24:26.05639425 +0000 UTC m=+414.178726170"
Sep 11 21:24:26.057322 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:26.057299    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ql5xg\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ql5xg"
Sep 11 21:24:26.077217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.077178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr" podStartSLOduration=2.07716574 podStartE2EDuration="2.07716574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:26.075845029 +0000 UTC m=+414.198176961" watchObservedRunningTime="2026-09-11 21:24:26.07716574 +0000 UTC m=+414.199497662"
Sep 11 21:24:26.078086 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:26.078062    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-bphzp\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-bphzp"
Sep 11 21:24:26.383381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.383307    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188e069f-2b20-4d23-b5bc-d39f70227358" path="/var/lib/kubelet/pods/188e069f-2b20-4d23-b5bc-d39f70227358/volumes"
Sep 11 21:24:26.383642 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:26.383629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac09c1d-4334-428f-b92f-a26d59e0899d" path="/var/lib/kubelet/pods/2ac09c1d-4334-428f-b92f-a26d59e0899d/volumes"
Sep 11 21:24:27.039893 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:27.039848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" event={"ID":"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f","Type":"ContainerStarted","Data":"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"}
Sep 11 21:24:27.040324 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:27.040164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:27.064628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:27.064554    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" podStartSLOduration=2.064540372 podStartE2EDuration="2.064540372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:27.064379135 +0000 UTC m=+415.186711056" watchObservedRunningTime="2026-09-11 21:24:27.064540372 +0000 UTC m=+415.186872293"
Sep 11 21:24:37.042244 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:37.042216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:37.042693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:37.042626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dpfwr"
Sep 11 21:24:38.047271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.047246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:24:38.116420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.116390    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:38.116645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.116603    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" podUID="27b19bf9-b070-42e5-af11-587f8d3b6290" containerName="manager" containerID="cri-o://5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72" gracePeriod=10
Sep 11 21:24:38.346676 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.346655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:38.466457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.466421    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69n2q\" (UniqueName: \"kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q\") pod \"27b19bf9-b070-42e5-af11-587f8d3b6290\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") "
Sep 11 21:24:38.466658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.466531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume\") pod \"27b19bf9-b070-42e5-af11-587f8d3b6290\" (UID: \"27b19bf9-b070-42e5-af11-587f8d3b6290\") "
Sep 11 21:24:38.466901 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.466878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "27b19bf9-b070-42e5-af11-587f8d3b6290" (UID: "27b19bf9-b070-42e5-af11-587f8d3b6290"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:24:38.468439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.468422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q" (OuterVolumeSpecName: "kube-api-access-69n2q") pod "27b19bf9-b070-42e5-af11-587f8d3b6290" (UID: "27b19bf9-b070-42e5-af11-587f8d3b6290"). InnerVolumeSpecName "kube-api-access-69n2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:24:38.567211 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.567181    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69n2q\" (UniqueName: \"kubernetes.io/projected/27b19bf9-b070-42e5-af11-587f8d3b6290-kube-api-access-69n2q\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:24:38.567211 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:38.567205    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27b19bf9-b070-42e5-af11-587f8d3b6290-extensions-socket-volume\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:24:39.093814 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.093780    2580 generic.go:356] "Generic (PLEG): container finished" podID="27b19bf9-b070-42e5-af11-587f8d3b6290" containerID="5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72" exitCode=0
Sep 11 21:24:39.094204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.093823    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"
Sep 11 21:24:39.094204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.093863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" event={"ID":"27b19bf9-b070-42e5-af11-587f8d3b6290","Type":"ContainerDied","Data":"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"}
Sep 11 21:24:39.094204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.093917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx" event={"ID":"27b19bf9-b070-42e5-af11-587f8d3b6290","Type":"ContainerDied","Data":"894ecef96f655cf3dc9b8c291c7ca6c90579a0d49835181c1291198ec8248038"}
Sep 11 21:24:39.094204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.093938    2580 scope.go:117] "RemoveContainer" containerID="5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"
Sep 11 21:24:39.102663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.102645    2580 scope.go:117] "RemoveContainer" containerID="5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"
Sep 11 21:24:39.102941 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:24:39.102922    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72\": container with ID starting with 5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72 not found: ID does not exist" containerID="5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"
Sep 11 21:24:39.103011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.102949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72"} err="failed to get container status \"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72\": rpc error: code = NotFound desc = could not find container \"5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72\": container with ID starting with 5487492914aec668b70fd3f9c57228f045f362dcb7e561f6e4aeb96f16660e72 not found: ID does not exist"
Sep 11 21:24:39.118166 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.118141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:39.125979 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:39.125956    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mgzdx"]
Sep 11 21:24:40.383361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:40.383330    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b19bf9-b070-42e5-af11-587f8d3b6290" path="/var/lib/kubelet/pods/27b19bf9-b070-42e5-af11-587f8d3b6290/volumes"
Sep 11 21:24:57.331294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.331199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:24:57.331725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.331555    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b19bf9-b070-42e5-af11-587f8d3b6290" containerName="manager"
Sep 11 21:24:57.331725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.331568    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b19bf9-b070-42e5-af11-587f8d3b6290" containerName="manager"
Sep 11 21:24:57.331725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.331628    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b19bf9-b070-42e5-af11-587f8d3b6290" containerName="manager"
Sep 11 21:24:57.333437 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.333422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.336432 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.336409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 21:24:57.336636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.336408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jdp7f\""
Sep 11 21:24:57.344662 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.344637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:24:57.373752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.373728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:24:57.416760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.416732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hmh\" (UniqueName: \"kubernetes.io/projected/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-kube-api-access-n4hmh\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.416903 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.416782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-config-file\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.517297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.517258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hmh\" (UniqueName: \"kubernetes.io/projected/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-kube-api-access-n4hmh\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.517438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.517330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-config-file\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.518005 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.517986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-config-file\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.527536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.527486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hmh\" (UniqueName: \"kubernetes.io/projected/ded0665b-ce16-48a1-aa5e-7a5c40b6c645-kube-api-access-n4hmh\") pod \"limitador-limitador-84994f7ffc-ss8j8\" (UID: \"ded0665b-ce16-48a1-aa5e-7a5c40b6c645\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.643395 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.643329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:57.913761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.913690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:24:57.916778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.916756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:24:57.917183 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.917145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:24:57.919278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.919256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92bp\" (UniqueName: \"kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp\") pod \"authorino-59c974bd48-lrbgx\" (UID: \"ca9a1f25-7cba-4e84-b030-3824bd6072cd\") " pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:24:57.919454 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.919437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nz7pm\""
Sep 11 21:24:57.920145 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:57.920124    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded0665b_ce16_48a1_aa5e_7a5c40b6c645.slice/crio-d97b63f64ba509be35979d8ad5e3f6002b70a474d1bb70927a25292b297cb197 WatchSource:0}: Error finding container d97b63f64ba509be35979d8ad5e3f6002b70a474d1bb70927a25292b297cb197: Status 404 returned error can't find the container with id d97b63f64ba509be35979d8ad5e3f6002b70a474d1bb70927a25292b297cb197
Sep 11 21:24:57.929066 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:57.929046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:24:58.020471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:58.020434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92bp\" (UniqueName: \"kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp\") pod \"authorino-59c974bd48-lrbgx\" (UID: \"ca9a1f25-7cba-4e84-b030-3824bd6072cd\") " pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:24:58.033420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:58.033397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92bp\" (UniqueName: \"kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp\") pod \"authorino-59c974bd48-lrbgx\" (UID: \"ca9a1f25-7cba-4e84-b030-3824bd6072cd\") " pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:24:58.178286 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:58.178203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8" event={"ID":"ded0665b-ce16-48a1-aa5e-7a5c40b6c645","Type":"ContainerStarted","Data":"d97b63f64ba509be35979d8ad5e3f6002b70a474d1bb70927a25292b297cb197"}
Sep 11 21:24:58.233735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:58.233690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:24:58.499024 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:58.499003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:24:58.501433 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:24:58.501415    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca9a1f25_7cba_4e84_b030_3824bd6072cd.slice/crio-5e68e76c9cc741a5b2b685fc7dd9c08fc9d96ebe1c1f1ed05ac5aad90ddf1c2b WatchSource:0}: Error finding container 5e68e76c9cc741a5b2b685fc7dd9c08fc9d96ebe1c1f1ed05ac5aad90ddf1c2b: Status 404 returned error can't find the container with id 5e68e76c9cc741a5b2b685fc7dd9c08fc9d96ebe1c1f1ed05ac5aad90ddf1c2b
Sep 11 21:24:59.184488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:59.184453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8" event={"ID":"ded0665b-ce16-48a1-aa5e-7a5c40b6c645","Type":"ContainerStarted","Data":"7b67b2229dc98f4f83d3a658764770bbb1fa5e6747605cbbb9690dd50ab54ece"}
Sep 11 21:24:59.184653 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:59.184576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:24:59.185720 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:59.185698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lrbgx" event={"ID":"ca9a1f25-7cba-4e84-b030-3824bd6072cd","Type":"ContainerStarted","Data":"5e68e76c9cc741a5b2b685fc7dd9c08fc9d96ebe1c1f1ed05ac5aad90ddf1c2b"}
Sep 11 21:24:59.201783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:24:59.201723    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8" podStartSLOduration=1.045233551 podStartE2EDuration="2.201703498s" totalImagesPullingTime="1.156469947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:59.201109981 +0000 UTC m=+447.323441902" watchObservedRunningTime="2026-09-11 21:24:59.201703498 +0000 UTC m=+447.324035421"
Sep 11 21:25:01.199547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:01.199481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lrbgx" event={"ID":"ca9a1f25-7cba-4e84-b030-3824bd6072cd","Type":"ContainerStarted","Data":"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"}
Sep 11 21:25:01.216405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:01.216360    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-lrbgx" podStartSLOduration=2.25004619 podStartE2EDuration="4.216344799s" totalImagesPullingTime="1.966298609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:25:01.215457962 +0000 UTC m=+449.337789883" watchObservedRunningTime="2026-09-11 21:25:01.216344799 +0000 UTC m=+449.338676717"
Sep 11 21:25:10.191626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:10.191596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"
Sep 11 21:25:23.264075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.264043    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-pvth2"]
Sep 11 21:25:23.270278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.270260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:23.275233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.275207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvth2"]
Sep 11 21:25:23.406737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.406683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfg54\" (UniqueName: \"kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54\") pod \"authorino-557f876f69-pvth2\" (UID: \"ce25183f-596f-4c67-8863-cfb8d390066f\") " pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:23.507490 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.507456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfg54\" (UniqueName: \"kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54\") pod \"authorino-557f876f69-pvth2\" (UID: \"ce25183f-596f-4c67-8863-cfb8d390066f\") " pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:23.510217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.510195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvth2"]
Sep 11 21:25:23.510416 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:25:23.510401    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jfg54], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-pvth2" podUID="ce25183f-596f-4c67-8863-cfb8d390066f"
Sep 11 21:25:23.519401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.519342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfg54\" (UniqueName: \"kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54\") pod \"authorino-557f876f69-pvth2\" (UID: \"ce25183f-596f-4c67-8863-cfb8d390066f\") " pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:23.541954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.541927    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:25:23.546159 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.546141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.548654 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.548637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 21:25:23.556439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.556419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:25:23.608262 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.608237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtqm\" (UniqueName: \"kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.608356 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.608301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.709273 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.709247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtqm\" (UniqueName: \"kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.709358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.709299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.711795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.711778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.721089 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.721066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtqm\" (UniqueName: \"kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm\") pod \"authorino-8676dd8574-q4mkj\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") " pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:23.856346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:23.856287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:25:24.130828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.130804    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:25:24.133020 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:25:24.132995    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59ba78f_672c_4fa2_b8db_fd8140cd7900.slice/crio-3f646cae921380d586b0a4a8a2de0434f299e397707930898d66de545c08c4bf WatchSource:0}: Error finding container 3f646cae921380d586b0a4a8a2de0434f299e397707930898d66de545c08c4bf: Status 404 returned error can't find the container with id 3f646cae921380d586b0a4a8a2de0434f299e397707930898d66de545c08c4bf
Sep 11 21:25:24.299735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.299708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:24.299735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.299725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8676dd8574-q4mkj" event={"ID":"c59ba78f-672c-4fa2-b8db-fd8140cd7900","Type":"ContainerStarted","Data":"3f646cae921380d586b0a4a8a2de0434f299e397707930898d66de545c08c4bf"}
Sep 11 21:25:24.309043 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.309024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:24.415739 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.415681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfg54\" (UniqueName: \"kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54\") pod \"ce25183f-596f-4c67-8863-cfb8d390066f\" (UID: \"ce25183f-596f-4c67-8863-cfb8d390066f\") "
Sep 11 21:25:24.417779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.417760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54" (OuterVolumeSpecName: "kube-api-access-jfg54") pod "ce25183f-596f-4c67-8863-cfb8d390066f" (UID: "ce25183f-596f-4c67-8863-cfb8d390066f"). InnerVolumeSpecName "kube-api-access-jfg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:25:24.516278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:24.516249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfg54\" (UniqueName: \"kubernetes.io/projected/ce25183f-596f-4c67-8863-cfb8d390066f-kube-api-access-jfg54\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:25:25.305656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.305627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvth2"
Sep 11 21:25:25.305656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.305651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8676dd8574-q4mkj" event={"ID":"c59ba78f-672c-4fa2-b8db-fd8140cd7900","Type":"ContainerStarted","Data":"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"}
Sep 11 21:25:25.324627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.324585    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8676dd8574-q4mkj" podStartSLOduration=1.79669529 podStartE2EDuration="2.324572467s" totalImagesPullingTime="527.877177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:25:25.323221475 +0000 UTC m=+473.445553395" watchObservedRunningTime="2026-09-11 21:25:25.324572467 +0000 UTC m=+473.446904387"
Sep 11 21:25:25.360077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.360041    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:25:25.360239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.360216    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-lrbgx" podUID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" containerName="authorino" containerID="cri-o://4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5" gracePeriod=30
Sep 11 21:25:25.365270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.365247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvth2"]
Sep 11 21:25:25.374710 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.374684    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvth2"]
Sep 11 21:25:25.606816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.606792    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:25:25.727113 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.727084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s92bp\" (UniqueName: \"kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp\") pod \"ca9a1f25-7cba-4e84-b030-3824bd6072cd\" (UID: \"ca9a1f25-7cba-4e84-b030-3824bd6072cd\") "
Sep 11 21:25:25.729255 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.729225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp" (OuterVolumeSpecName: "kube-api-access-s92bp") pod "ca9a1f25-7cba-4e84-b030-3824bd6072cd" (UID: "ca9a1f25-7cba-4e84-b030-3824bd6072cd"). InnerVolumeSpecName "kube-api-access-s92bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:25:25.828553 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:25.828495    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s92bp\" (UniqueName: \"kubernetes.io/projected/ca9a1f25-7cba-4e84-b030-3824bd6072cd-kube-api-access-s92bp\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:25:26.311253 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.311218    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" containerID="4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5" exitCode=0
Sep 11 21:25:26.311675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.311261    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lrbgx"
Sep 11 21:25:26.311675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.311308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lrbgx" event={"ID":"ca9a1f25-7cba-4e84-b030-3824bd6072cd","Type":"ContainerDied","Data":"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"}
Sep 11 21:25:26.311675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.311348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lrbgx" event={"ID":"ca9a1f25-7cba-4e84-b030-3824bd6072cd","Type":"ContainerDied","Data":"5e68e76c9cc741a5b2b685fc7dd9c08fc9d96ebe1c1f1ed05ac5aad90ddf1c2b"}
Sep 11 21:25:26.311675 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.311368    2580 scope.go:117] "RemoveContainer" containerID="4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"
Sep 11 21:25:26.320156 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.320145    2580 scope.go:117] "RemoveContainer" containerID="4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"
Sep 11 21:25:26.320432 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:25:26.320415    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5\": container with ID starting with 4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5 not found: ID does not exist" containerID="4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"
Sep 11 21:25:26.320480 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.320440    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5"} err="failed to get container status \"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5\": rpc error: code = NotFound desc = could not find container \"4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5\": container with ID starting with 4b33c89fac03da6f1a47b5e40c7f3d31f41738285392095f8f1e063b6a7ebcf5 not found: ID does not exist"
Sep 11 21:25:26.332130 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.332107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:25:26.337075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.337056    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lrbgx"]
Sep 11 21:25:26.382337 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.382317    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" path="/var/lib/kubelet/pods/ca9a1f25-7cba-4e84-b030-3824bd6072cd/volumes"
Sep 11 21:25:26.382673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:25:26.382660    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce25183f-596f-4c67-8863-cfb8d390066f" path="/var/lib/kubelet/pods/ce25183f-596f-4c67-8863-cfb8d390066f/volumes"
Sep 11 21:26:09.265301 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:26:09.265264    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:26:45.604467 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:26:45.604395    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:26:51.298768 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:26:51.298725    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:26:57.196362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:26:57.196326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:27:01.699991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:01.699951    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:27:15.294985 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.294953    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"]
Sep 11 21:27:15.295338 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.295275    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" containerName="authorino"
Sep 11 21:27:15.295338 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.295286    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" containerName="authorino"
Sep 11 21:27:15.295410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.295358    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9a1f25-7cba-4e84-b030-3824bd6072cd" containerName="authorino"
Sep 11 21:27:15.298527 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.298493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.301217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.301193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 21:27:15.302580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.302557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 21:27:15.302679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.302580    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hc7mn\""
Sep 11 21:27:15.302679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.302582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 21:27:15.310190 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.310169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"]
Sep 11 21:27:15.422378 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6822e42-980d-404d-b371-0c288e6fe056-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.422545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.422545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.422545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.422545 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.422678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.422615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9rm\" (UniqueName: \"kubernetes.io/projected/f6822e42-980d-404d-b371-0c288e6fe056-kube-api-access-xd9rm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9rm\" (UniqueName: \"kubernetes.io/projected/f6822e42-980d-404d-b371-0c288e6fe056-kube-api-access-xd9rm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6822e42-980d-404d-b371-0c288e6fe056-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.523853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.523834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.524236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.524217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.524315 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.524279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.524368 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.524321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.526184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.526163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6822e42-980d-404d-b371-0c288e6fe056-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.526348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.526332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6822e42-980d-404d-b371-0c288e6fe056-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.532887 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.532866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9rm\" (UniqueName: \"kubernetes.io/projected/f6822e42-980d-404d-b371-0c288e6fe056-kube-api-access-xd9rm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn\" (UID: \"f6822e42-980d-404d-b371-0c288e6fe056\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.610232 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.610175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:15.883997 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:15.883974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"]
Sep 11 21:27:15.886561 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:27:15.886535    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6822e42_980d_404d_b371_0c288e6fe056.slice/crio-bb1b0ba3ec43a9e2c09c1905dcc92e8a9a23193c42739c042941bdc61d87d7cd WatchSource:0}: Error finding container bb1b0ba3ec43a9e2c09c1905dcc92e8a9a23193c42739c042941bdc61d87d7cd: Status 404 returned error can't find the container with id bb1b0ba3ec43a9e2c09c1905dcc92e8a9a23193c42739c042941bdc61d87d7cd
Sep 11 21:27:16.806465 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:16.806431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn" event={"ID":"f6822e42-980d-404d-b371-0c288e6fe056","Type":"ContainerStarted","Data":"bb1b0ba3ec43a9e2c09c1905dcc92e8a9a23193c42739c042941bdc61d87d7cd"}
Sep 11 21:27:16.905756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:16.905730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:27:17.812713 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:17.812682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn" event={"ID":"f6822e42-980d-404d-b371-0c288e6fe056","Type":"ContainerStarted","Data":"d45ec2b4e5b3d98f261b11ac4f856cb21b51f0387f265d23d05c12c8a65436c4"}
Sep 11 21:27:17.844283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:17.844231    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn" podStartSLOduration=1.874872489 podStartE2EDuration="2.844214345s" totalImagesPullingTime="969.341856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:17.843085598 +0000 UTC m=+585.965417521" watchObservedRunningTime="2026-09-11 21:27:17.844214345 +0000 UTC m=+585.966546266"
Sep 11 21:27:25.611176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:25.611138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:25.611609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:25.611210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:25.623260 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:25.623238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:25.861863 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:25.861782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn"
Sep 11 21:27:30.338733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:30.338636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:27:45.741372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.741341    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"]
Sep 11 21:27:45.744359 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.744344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.748753 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:27:45.748728    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="98453237-c6e0-4e55-88cf-b55181507933" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.749836 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:27:45.749813    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"premium-simulated-simulated-premium-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 11 21:27:45.790062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.790035    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"]
Sep 11 21:27:45.855229 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.855326 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.855367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.855413 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64k5w\" (UniqueName: \"kubernetes.io/projected/98453237-c6e0-4e55-88cf-b55181507933-kube-api-access-64k5w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.855489 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.855583 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.855545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.956871 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.956839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.956883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.956933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.956973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64k5w\" (UniqueName: \"kubernetes.io/projected/98453237-c6e0-4e55-88cf-b55181507933-kube-api-access-64k5w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.957000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.957047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.957208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957360 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.957270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.957422 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.957384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.959372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.959355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98453237-c6e0-4e55-88cf-b55181507933-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:45.969730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:45.969706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64k5w\" (UniqueName: \"kubernetes.io/projected/98453237-c6e0-4e55-88cf-b55181507933-kube-api-access-64k5w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:46.957407 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:27:46.957380    2580 secret.go:189] Couldn't get secret llm/premium-simulated-simulated-premium-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 11 21:27:46.957786 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:27:46.957460    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs podName:98453237-c6e0-4e55-88cf-b55181507933 nodeName:}" failed. No retries permitted until 2026-09-11 21:27:47.457440427 +0000 UTC m=+615.579772334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs") pod "premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt" (UID: "98453237-c6e0-4e55-88cf-b55181507933") : failed to sync secret cache: timed out waiting for the condition
Sep 11 21:27:47.074102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.074073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 21:27:47.468380 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.468343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:47.473622 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.471650    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98453237-c6e0-4e55-88cf-b55181507933-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt\" (UID: \"98453237-c6e0-4e55-88cf-b55181507933\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:47.553524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.553476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:47.835473 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.835442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"]
Sep 11 21:27:47.839736 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:27:47.839712    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98453237_c6e0_4e55_88cf_b55181507933.slice/crio-49adb8b1389eefa0b19860cae37ec0ca20e5e9329ddca653967aba9c8d7ce873 WatchSource:0}: Error finding container 49adb8b1389eefa0b19860cae37ec0ca20e5e9329ddca653967aba9c8d7ce873: Status 404 returned error can't find the container with id 49adb8b1389eefa0b19860cae37ec0ca20e5e9329ddca653967aba9c8d7ce873
Sep 11 21:27:47.841213 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.841191    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:27:47.943066 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:47.943036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt" event={"ID":"98453237-c6e0-4e55-88cf-b55181507933","Type":"ContainerStarted","Data":"49adb8b1389eefa0b19860cae37ec0ca20e5e9329ddca653967aba9c8d7ce873"}
Sep 11 21:27:48.026777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:48.026749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:27:48.949450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:48.949404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt" event={"ID":"98453237-c6e0-4e55-88cf-b55181507933","Type":"ContainerStarted","Data":"25c473b5abff3e1709383a88cfac71997057b99cdf4d784839d33b386ef7722f"}
Sep 11 21:27:57.553718 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:57.553644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:57.553718 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:57.553690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:57.565849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:57.565820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:27:57.600012 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:57.599970    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt" podStartSLOduration=12.422071435 podStartE2EDuration="12.599955176s" totalImagesPullingTime="177.883741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:49.023036872 +0000 UTC m=+617.145368771" watchObservedRunningTime="2026-09-11 21:27:57.599955176 +0000 UTC m=+625.722287113"
Sep 11 21:27:57.998788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:27:57.998763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt"
Sep 11 21:29:02.913089 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:02.913059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f66bb765f-gbm9z"]
Sep 11 21:29:02.916152 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:02.916136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:02.941440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:02.941414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f66bb765f-gbm9z"]
Sep 11 21:29:03.004593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.004572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvls\" (UniqueName: \"kubernetes.io/projected/5987d22f-1b14-4211-a1e3-a7f7be363569-kube-api-access-nrvls\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.004703 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.004627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5987d22f-1b14-4211-a1e3-a7f7be363569-tls-cert\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.105677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.105653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvls\" (UniqueName: \"kubernetes.io/projected/5987d22f-1b14-4211-a1e3-a7f7be363569-kube-api-access-nrvls\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.105773 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.105726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5987d22f-1b14-4211-a1e3-a7f7be363569-tls-cert\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.108282 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.108254    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5987d22f-1b14-4211-a1e3-a7f7be363569-tls-cert\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.115435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.115410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvls\" (UniqueName: \"kubernetes.io/projected/5987d22f-1b14-4211-a1e3-a7f7be363569-kube-api-access-nrvls\") pod \"authorino-5f66bb765f-gbm9z\" (UID: \"5987d22f-1b14-4211-a1e3-a7f7be363569\") " pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.224352 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.224285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f66bb765f-gbm9z"
Sep 11 21:29:03.494239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:03.494218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f66bb765f-gbm9z"]
Sep 11 21:29:03.496130 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:03.496104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5987d22f_1b14_4211_a1e3_a7f7be363569.slice/crio-9d9f47b4f776ed394c158fd314b26c00b3e36ee7f2cba4c6cc967a320430844a WatchSource:0}: Error finding container 9d9f47b4f776ed394c158fd314b26c00b3e36ee7f2cba4c6cc967a320430844a: Status 404 returned error can't find the container with id 9d9f47b4f776ed394c158fd314b26c00b3e36ee7f2cba4c6cc967a320430844a
Sep 11 21:29:04.274353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.274315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f66bb765f-gbm9z" event={"ID":"5987d22f-1b14-4211-a1e3-a7f7be363569","Type":"ContainerStarted","Data":"e399e3f5701311b418a6784d0dabc2eeebd800bc199ff677fd008e71c387894e"}
Sep 11 21:29:04.274752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.274364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f66bb765f-gbm9z" event={"ID":"5987d22f-1b14-4211-a1e3-a7f7be363569","Type":"ContainerStarted","Data":"9d9f47b4f776ed394c158fd314b26c00b3e36ee7f2cba4c6cc967a320430844a"}
Sep 11 21:29:04.292220 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.292177    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f66bb765f-gbm9z" podStartSLOduration=1.684514975 podStartE2EDuration="2.292159689s" totalImagesPullingTime="607.644714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:04.291490324 +0000 UTC m=+692.413822244" watchObservedRunningTime="2026-09-11 21:29:04.292159689 +0000 UTC m=+692.414491609"
Sep 11 21:29:04.327547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.327513    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:29:04.327733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.327688    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8676dd8574-q4mkj" podUID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" containerName="authorino" containerID="cri-o://4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538" gracePeriod=30
Sep 11 21:29:04.578156 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.578122    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:29:04.615729 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.615646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert\") pod \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") "
Sep 11 21:29:04.615729 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.615719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwtqm\" (UniqueName: \"kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm\") pod \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\" (UID: \"c59ba78f-672c-4fa2-b8db-fd8140cd7900\") "
Sep 11 21:29:04.617822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.617784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm" (OuterVolumeSpecName: "kube-api-access-xwtqm") pod "c59ba78f-672c-4fa2-b8db-fd8140cd7900" (UID: "c59ba78f-672c-4fa2-b8db-fd8140cd7900"). InnerVolumeSpecName "kube-api-access-xwtqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:04.625793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.625768    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c59ba78f-672c-4fa2-b8db-fd8140cd7900" (UID: "c59ba78f-672c-4fa2-b8db-fd8140cd7900"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:29:04.716640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.716605    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtqm\" (UniqueName: \"kubernetes.io/projected/c59ba78f-672c-4fa2-b8db-fd8140cd7900-kube-api-access-xwtqm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:29:04.716640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:04.716631    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c59ba78f-672c-4fa2-b8db-fd8140cd7900-tls-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:29:05.282395 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.282352    2580 generic.go:356] "Generic (PLEG): container finished" podID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" containerID="4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538" exitCode=0
Sep 11 21:29:05.283082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.283058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8676dd8574-q4mkj"
Sep 11 21:29:05.285270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.285207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8676dd8574-q4mkj" event={"ID":"c59ba78f-672c-4fa2-b8db-fd8140cd7900","Type":"ContainerDied","Data":"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"}
Sep 11 21:29:05.285400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.285294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8676dd8574-q4mkj" event={"ID":"c59ba78f-672c-4fa2-b8db-fd8140cd7900","Type":"ContainerDied","Data":"3f646cae921380d586b0a4a8a2de0434f299e397707930898d66de545c08c4bf"}
Sep 11 21:29:05.285400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.285332    2580 scope.go:117] "RemoveContainer" containerID="4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"
Sep 11 21:29:05.301226 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.301208    2580 scope.go:117] "RemoveContainer" containerID="4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"
Sep 11 21:29:05.301697 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:29:05.301548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538\": container with ID starting with 4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538 not found: ID does not exist" containerID="4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"
Sep 11 21:29:05.301697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.301582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538"} err="failed to get container status \"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538\": rpc error: code = NotFound desc = could not find container \"4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538\": container with ID starting with 4d5116b459acd61a1df7e0cf952c5310072366d0d0a2ba38747d0b9ba9aa6538 not found: ID does not exist"
Sep 11 21:29:05.311353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.311331    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:29:05.318474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:05.318451    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8676dd8574-q4mkj"]
Sep 11 21:29:06.383618 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:06.383591    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" path="/var/lib/kubelet/pods/c59ba78f-672c-4fa2-b8db-fd8140cd7900/volumes"
Sep 11 21:29:22.201808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.201777    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:29:22.202240 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.202225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" containerName="authorino"
Sep 11 21:29:22.202333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.202243    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" containerName="authorino"
Sep 11 21:29:22.202372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.202333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59ba78f-672c-4fa2-b8db-fd8140cd7900" containerName="authorino"
Sep 11 21:29:22.206955 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.206936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.210576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.210559    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1a1e3a-openshift-default-dockercfg-sd2gt\""
Sep 11 21:29:22.236353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.236319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:29:22.239673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.239793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.239793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.239793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.239793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.240145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwfv\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.240145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.240145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.239976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.240145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.240010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwfv\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.341861 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.342054 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.341980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.342054 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.342027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.342380 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.342359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.342575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.342400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.342984 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.342959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.344165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.344142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.344391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.344373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.366708 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.366686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.378691 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.378667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwfv\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv\") pod \"e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") " pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.505989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.505960    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:29:22.510658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.510638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.514216 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.514186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-83ba87-openshift-default-dockercfg-rdvf7\""
Sep 11 21:29:22.517333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.517313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:22.528621 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.528537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:29:22.543829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.543801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544009 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.543991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brbc\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544664 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.544892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.544880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.645998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.645929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.645998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.645985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brbc\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.646233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.647239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.646951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.647239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.647168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.647239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.647228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.647534 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.647349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.647534 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.647395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.649584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.649554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.649692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.649649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.656994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.656972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.657460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.657444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brbc\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc\") pod \"e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") " pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:22.816933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.816911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:29:22.818827 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:22.818802    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eba0fd_0581_41cd_bd97_2c76791bbf25.slice/crio-da8a0f6cadc6e6facc9380e0f62b4af23bb3ea01befa66df2df6d56d58631c47 WatchSource:0}: Error finding container da8a0f6cadc6e6facc9380e0f62b4af23bb3ea01befa66df2df6d56d58631c47: Status 404 returned error can't find the container with id da8a0f6cadc6e6facc9380e0f62b4af23bb3ea01befa66df2df6d56d58631c47
Sep 11 21:29:22.821132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.821097    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:22.821241 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.821174    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:22.821241 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.821220    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:22.822079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:22.822062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:23.123207 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:23.123176    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a75ab54_e867_4ade_8663_a176c9b6ea25.slice/crio-ae20b0cb1ee5c4f168d03913440a6a68feaccd388d4947b590adc8966667d28f WatchSource:0}: Error finding container ae20b0cb1ee5c4f168d03913440a6a68feaccd388d4947b590adc8966667d28f: Status 404 returned error can't find the container with id ae20b0cb1ee5c4f168d03913440a6a68feaccd388d4947b590adc8966667d28f
Sep 11 21:29:23.124442 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.124420    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:29:23.125846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.125811    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:23.125921 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.125907    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:23.125971 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.125937    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:23.373603 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.373511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerStarted","Data":"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"}
Sep 11 21:29:23.373603 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.373600    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerStarted","Data":"ae20b0cb1ee5c4f168d03913440a6a68feaccd388d4947b590adc8966667d28f"}
Sep 11 21:29:23.375264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.375234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerStarted","Data":"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"}
Sep 11 21:29:23.375375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.375284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerStarted","Data":"da8a0f6cadc6e6facc9380e0f62b4af23bb3ea01befa66df2df6d56d58631c47"}
Sep 11 21:29:23.402063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.402021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podStartSLOduration=1.402006122 podStartE2EDuration="1.402006122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:23.400969491 +0000 UTC m=+711.523301410" watchObservedRunningTime="2026-09-11 21:29:23.402006122 +0000 UTC m=+711.524338094"
Sep 11 21:29:23.431977 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.431921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podStartSLOduration=1.431902685 podStartE2EDuration="1.431902685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:23.429900697 +0000 UTC m=+711.552232645" watchObservedRunningTime="2026-09-11 21:29:23.431902685 +0000 UTC m=+711.554234605"
Sep 11 21:29:23.518597 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.518533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:23.523181 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.523159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:23.822658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:23.822630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:24.383776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:24.383747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:24.384143 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:24.383818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:24.827258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:24.827232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:25.384846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:25.384810    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:25.385616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:25.385600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:25.614058 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:25.614023    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:29:31.442289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.442260    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:29:31.445702 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.445685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.450164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.450146    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1e9b41-openshift-default-dockercfg-q6j2r\""
Sep 11 21:29:31.523288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523451 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523493 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpknn\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.523640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.523635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.528226 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.528205    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:29:31.624817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.624794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.624924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.624820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.624924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.624836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpknn\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.624924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.624865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625083 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.624985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625083 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625083 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.625767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.625719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.626221 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.626197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.627447 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.627426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.627728 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.627711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.640230 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.640208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpknn\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.659175 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.659150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token\") pod \"e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") " pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:31.755544 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:31.755524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:32.061998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.061972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:29:32.063319 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:32.063288    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0c74ed_33cb_4d97_929b_b417861f5f03.slice/crio-3d9d9baeee00713f85c2fdef072a82a94643dcf6d82d7c96969fddd044705ad6 WatchSource:0}: Error finding container 3d9d9baeee00713f85c2fdef072a82a94643dcf6d82d7c96969fddd044705ad6: Status 404 returned error can't find the container with id 3d9d9baeee00713f85c2fdef072a82a94643dcf6d82d7c96969fddd044705ad6
Sep 11 21:29:32.067542 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.066327    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:32.067542 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.066478    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:32.072818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.072785    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:32.422904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.422872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerStarted","Data":"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"}
Sep 11 21:29:32.423055 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.422922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerStarted","Data":"3d9d9baeee00713f85c2fdef072a82a94643dcf6d82d7c96969fddd044705ad6"}
Sep 11 21:29:32.455979 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.455931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podStartSLOduration=1.455915179 podStartE2EDuration="1.455915179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:32.451552131 +0000 UTC m=+720.573884052" watchObservedRunningTime="2026-09-11 21:29:32.455915179 +0000 UTC m=+720.578247096"
Sep 11 21:29:32.756009 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:32.755979    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:33.762047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:33.762017    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:34.433462 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:34.433426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:34.434853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:34.434833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:29:35.204661 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.204635    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:29:35.210405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.210380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.219759 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.219315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b37c5b2-openshift-default-dockercfg-97pd2\""
Sep 11 21:29:35.249529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.241452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:29:35.268398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268716 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268716 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7b8\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.268948 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.268826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369436 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7b8\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.369961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.369919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.370459 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.370430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.370784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.370486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.370784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.370703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.370925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.370907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.372743 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.372720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.372981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.372965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.379800 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.379772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7b8\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.379986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.379971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token\") pod \"e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") " pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.544376 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.544332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:35.886395 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.886372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:29:35.888004 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:35.887964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d518ee7_e1fd_4be5_9602_c6e50b97226d.slice/crio-eb470c00c5f8ee9d3e4bfd19ebd4413358ac15ba6beb4d5f40cf164f6be46fe1 WatchSource:0}: Error finding container eb470c00c5f8ee9d3e4bfd19ebd4413358ac15ba6beb4d5f40cf164f6be46fe1: Status 404 returned error can't find the container with id eb470c00c5f8ee9d3e4bfd19ebd4413358ac15ba6beb4d5f40cf164f6be46fe1
Sep 11 21:29:35.890898 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.890855    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:35.891000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.890946    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:35.891000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:35.890987    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:36.450155 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.450116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerStarted","Data":"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"}
Sep 11 21:29:36.450625 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.450175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerStarted","Data":"eb470c00c5f8ee9d3e4bfd19ebd4413358ac15ba6beb4d5f40cf164f6be46fe1"}
Sep 11 21:29:36.477332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.477278    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podStartSLOduration=1.477260363 podStartE2EDuration="1.477260363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:36.47632682 +0000 UTC m=+724.598658740" watchObservedRunningTime="2026-09-11 21:29:36.477260363 +0000 UTC m=+724.599592289"
Sep 11 21:29:36.544778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.544712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:36.553124 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.552925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:36.668365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.668332    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:29:36.672439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.672416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.675197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.675174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1e9b41-serving-cert\""
Sep 11 21:29:36.675299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.675283    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czlrt\""
Sep 11 21:29:36.675558 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.675538    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:29:36.675697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.675562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:29:36.675783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.675766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:29:36.683281 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.683263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:29:36.783803 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.783775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.783986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.783838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.783986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.783861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.783986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.783895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccnjb\" (UniqueName: \"kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.885033 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.884998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.885274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.885253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.885416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.885399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccnjb\" (UniqueName: \"kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.885628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.885611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.886367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.886346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.887749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.887731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.887850 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.887790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.895072 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.895051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccnjb\" (UniqueName: \"kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb\") pod \"maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:36.987280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:36.987248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:37.296339 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:37.296301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:29:37.300094 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:37.300073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff3303a_fb03_4188_a6d2_be32fdf4305a.slice/crio-06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e WatchSource:0}: Error finding container 06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e: Status 404 returned error can't find the container with id 06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e
Sep 11 21:29:37.465613 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:37.465481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" event={"ID":"7ff3303a-fb03-4188-a6d2-be32fdf4305a","Type":"ContainerStarted","Data":"06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e"}
Sep 11 21:29:37.465983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:37.465825    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:37.467041 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:37.467023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:29:40.487096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:40.486146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" event={"ID":"7ff3303a-fb03-4188-a6d2-be32fdf4305a","Type":"ContainerStarted","Data":"2b7a46ceaa57be68f1277c7ec46e35b9a5a548271ea99b01d26952d79e4a4ebd"}
Sep 11 21:29:40.487096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:40.486251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:40.517204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:40.517157    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" podStartSLOduration=1.597554201 podStartE2EDuration="4.517142516s" totalImagesPullingTime="2.919588315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:40.516327052 +0000 UTC m=+728.638658974" watchObservedRunningTime="2026-09-11 21:29:40.517142516 +0000 UTC m=+728.639474436"
Sep 11 21:29:41.426716 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:41.426678    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 21:29:41.426888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:41.426747    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 21:29:43.503811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.503783    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerID="2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e" exitCode=0
Sep 11 21:29:43.504170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.503859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerDied","Data":"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"}
Sep 11 21:29:43.504275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.504259    2580 scope.go:117] "RemoveContainer" containerID="2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"
Sep 11 21:29:43.505326 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.505301    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:43.505385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.505372    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:43.505418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:43.505403    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:44.510438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:44.510405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerStarted","Data":"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"}
Sep 11 21:29:44.623981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:44.623942    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:29:44.823184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:44.823109    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:44.823365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:44.823339    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:44.823426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:44.823404    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:45.823317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:45.823274    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:45.823797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:45.823327    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:46.360447 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.360412    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:29:46.368287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.368265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.373777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.373758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-02c60f-openshift-default-dockercfg-7vk4h\""
Sep 11 21:29:46.393060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.393040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:29:46.495769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfxt\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.495967 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.495944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.497945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.497924    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:29:46.597299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfxt\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.597981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.597710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598203 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598378 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598486 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.598732 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.598646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.599159 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.599134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.600856 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.600832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.601170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.601146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.611495 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.611444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.615287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.615268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfxt\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt\") pod \"e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") " pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.677883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.677841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:46.822587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.822555    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:46.822756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:46.822625    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:47.002484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.002452    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:29:47.006513 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:29:47.006470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecb8aa0_11fa_49bd_819f_b6a9d2f1396c.slice/crio-08d5cbf09b299b169fe1baac2da4eb208c9471d2fde9b580617c6654683d930c WatchSource:0}: Error finding container 08d5cbf09b299b169fe1baac2da4eb208c9471d2fde9b580617c6654683d930c: Status 404 returned error can't find the container with id 08d5cbf09b299b169fe1baac2da4eb208c9471d2fde9b580617c6654683d930c
Sep 11 21:29:47.008888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.008855    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:47.008981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.008923    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:47.008981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.008950    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:47.530185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.530151    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerStarted","Data":"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"}
Sep 11 21:29:47.530520 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.530464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerStarted","Data":"08d5cbf09b299b169fe1baac2da4eb208c9471d2fde9b580617c6654683d930c"}
Sep 11 21:29:47.620453 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.620381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podStartSLOduration=1.620359484 podStartE2EDuration="1.620359484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:47.61579566 +0000 UTC m=+735.738127580" watchObservedRunningTime="2026-09-11 21:29:47.620359484 +0000 UTC m=+735.742691404"
Sep 11 21:29:47.678177 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.678135    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:47.823685 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.823571    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:47.823964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:47.823912    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:48.685841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:48.685808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:48.822944 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:48.822902    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:48.823266 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:48.823236    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:49.547538 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:49.547487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:49.555526 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:49.553260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:29:49.824949 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:49.824868    2580 patch_prober.go:28] interesting pod/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 21:29:49.825466 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:49.825434    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 21:29:50.826990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:50.826969    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:50.827899 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:50.827853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:50.828919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:50.828890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:29:55.423768 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:55.423670    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 21:29:55.423768 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:55.423753    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.595194    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerID="26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1" exitCode=0
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.595267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerDied","Data":"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"}
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.595675    2580 scope.go:117] "RemoveContainer" containerID="26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.596728    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.596790    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:57.772816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:57.596822    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.601831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.601746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerStarted","Data":"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"}
Sep 11 21:29:58.603363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.603328    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerID="52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734" exitCode=0
Sep 11 21:29:58.603493 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.603401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerDied","Data":"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"}
Sep 11 21:29:58.603778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.603760    2580 scope.go:117] "RemoveContainer" containerID="52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"
Sep 11 21:29:58.604922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.604893    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.604999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.604972    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.605043 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.605009    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.605079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.605061    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerID="b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189" exitCode=0
Sep 11 21:29:58.605136 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.605119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerDied","Data":"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"}
Sep 11 21:29:58.605426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.605415    2580 scope.go:117] "RemoveContainer" containerID="b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"
Sep 11 21:29:58.606429 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.606403    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.606485 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.606461    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:58.606552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:58.606489    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:29:59.518387 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.518353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:29:59.518763 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.518547    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:29:59.518763 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.518592    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:29:59.611756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.611722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerStarted","Data":"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"}
Sep 11 21:29:59.613710 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.613686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerStarted","Data":"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"}
Sep 11 21:29:59.756904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.756872    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:29:59.757188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.757149    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:29:59.757342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:29:59.757323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:30:00.217024 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.216979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"]
Sep 11 21:30:00.221295 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.221271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"
Sep 11 21:30:00.222445 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.222419    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"]
Sep 11 21:30:00.224683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.224667    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nhbxk\""
Sep 11 21:30:00.225058 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.225040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"
Sep 11 21:30:00.235321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.234731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"
Sep 11 21:30:00.245717 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.245556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"
Sep 11 21:30:00.274382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.274347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"]
Sep 11 21:30:00.275593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.275569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"]
Sep 11 21:30:00.293457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.293419    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"]
Sep 11 21:30:00.315599 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.314339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"
Sep 11 21:30:00.340529 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.338799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"]
Sep 11 21:30:00.468710 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.468418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"
Sep 11 21:30:00.521536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.518870    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:30:00.521536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.518931    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:30:00.546917 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.546870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:30:00.547085 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.547010    2580 patch_prober.go:28] interesting pod/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:30:00.547085 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.547068    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 21:30:00.757827 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.756909    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:00.757827 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.756970    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:00.843626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:00.843592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"]
Sep 11 21:30:01.025655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.015866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"]
Sep 11 21:30:01.203039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.202980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"]
Sep 11 21:30:01.518145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.518088    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:30:01.518333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.518166    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:30:01.553600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.553552    2580 patch_prober.go:28] interesting pod/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:30:01.554055 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.553624    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 21:30:01.708249 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.708211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9" event={"ID":"201f7648-1d93-4477-9cd1-5a3d67e39c5d","Type":"ContainerStarted","Data":"bfe097aaff656c5236521bd3c494cd81c9812f6588cc53f3f4b5759c9dcc6546"}
Sep 11 21:30:01.710568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.710534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92" event={"ID":"d9f4a154-6bc2-4db5-af83-c9205d1accb0","Type":"ContainerStarted","Data":"703101bdebe4f4ffd6173586dda7a1c9b53637334fe74a6ee520827d7cd77289"}
Sep 11 21:30:01.712386 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.712364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2" event={"ID":"2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15","Type":"ContainerStarted","Data":"ade7b9b6a5680a2861621d38508475819957fbb3cb19ac7c1fd14b2c37505cf5"}
Sep 11 21:30:01.756630 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.756594    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:01.756904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:01.756874    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:02.521596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.518038    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:30:02.521596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.518102    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:30:02.553568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.551634    2580 patch_prober.go:28] interesting pod/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:30:02.553568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.551694    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 21:30:02.756558 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.755907    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:02.756558 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:02.755972    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:03.518527 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.518026    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:30:03.518527 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.518091    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:30:03.545524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.544862    2580 patch_prober.go:28] interesting pod/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 21:30:03.545524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.544939    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 21:30:03.758344 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.757917    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:03.758344 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:03.757981    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:04.435976 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.435945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:30:04.520528 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.519388    2580 patch_prober.go:28] interesting pod/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 11 21:30:04.520528 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.519445    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 11 21:30:04.524539 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.521930    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:30:04.524539 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.522133    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy" containerID="cri-o://86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573" gracePeriod=30
Sep 11 21:30:04.760920 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.760859    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:04.761374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:04.760926    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:05.551525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.548778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:30:05.551525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.550203    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:30:05.552845 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.552678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:30:05.628915 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.628581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:05.637384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.637350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.644165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.644133    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d3e394ce-gw-openshift-default-dockercfg-fst5q\""
Sep 11 21:30:05.655104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.655068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:05.680953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.680920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681345 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxms\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681520 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681696 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.681983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.681968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.682093 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.682080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.682240 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.682226    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.764530 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.758371    2580 patch_prober.go:28] interesting pod/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 11 21:30:05.764530 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.758468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 11 21:30:05.785239 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.784496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.785510 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.785467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxms\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.790668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.790638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.791416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.791247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.791714 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.791694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.791864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.791841    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.792000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.791724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.792130 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.792113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.792345 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.792326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.792474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.792459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.792989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.792970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.793322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.793287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.793852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.793833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.794286 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.794269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.796554 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.795026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.796983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.796964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.809719 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.809616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxms\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.812016 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.811972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token\") pod \"e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") " pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:05.989077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:05.989038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:06.418764 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.418085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:06.420066 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:30:06.420037    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2b07ec_61ec_42c3_a24f_8e37b571aa4a.slice/crio-5f8936dd903fbfe9dd62298ebe4ffa6f8dcadd1994519968676e18ef1e69aac9 WatchSource:0}: Error finding container 5f8936dd903fbfe9dd62298ebe4ffa6f8dcadd1994519968676e18ef1e69aac9: Status 404 returned error can't find the container with id 5f8936dd903fbfe9dd62298ebe4ffa6f8dcadd1994519968676e18ef1e69aac9
Sep 11 21:30:06.424355 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.424318    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:06.424433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.424403    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:06.424521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.424442    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:06.787589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.787047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerStarted","Data":"c686e86094ef33509b053d8d535ba3d5c6f1029b1d138ec32f922b06ff612857"}
Sep 11 21:30:06.787589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.787108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerStarted","Data":"5f8936dd903fbfe9dd62298ebe4ffa6f8dcadd1994519968676e18ef1e69aac9"}
Sep 11 21:30:06.815152 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.815086    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podStartSLOduration=1.815064013 podStartE2EDuration="1.815064013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:06.811188125 +0000 UTC m=+754.933520048" watchObservedRunningTime="2026-09-11 21:30:06.815064013 +0000 UTC m=+754.937395934"
Sep 11 21:30:06.911491 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.911457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:30:06.913537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.913484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:30:06.995982 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:06.995895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:07.466034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:07.465951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:30:07.995275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:07.995249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:08.799317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.799257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9" event={"ID":"201f7648-1d93-4477-9cd1-5a3d67e39c5d","Type":"ContainerStarted","Data":"fd21884ef99ba1a9bc4018d5d5323c964c21d932dda076ed8c6ac76753a186c3"}
Sep 11 21:30:08.800921 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.800895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92" event={"ID":"d9f4a154-6bc2-4db5-af83-c9205d1accb0","Type":"ContainerStarted","Data":"2c1bd2400fed78efe7584d30829a770ceed75cb76a46a7e58fe5f0d9a0b82701"}
Sep 11 21:30:08.802431 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.802410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2" event={"ID":"2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15","Type":"ContainerStarted","Data":"26078ba125765f31b8eb7f3a94dfaa7e9f0506e7a85e9185bbde21bac4bb5c5b"}
Sep 11 21:30:08.802725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.802708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:08.803708 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.803692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:08.820168 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.820119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9" podStartSLOduration=1.7244408789999999 podStartE2EDuration="8.820102427s" totalImagesPullingTime="7.095661548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:08.817619282 +0000 UTC m=+756.939951203" watchObservedRunningTime="2026-09-11 21:30:08.820102427 +0000 UTC m=+756.942434348"
Sep 11 21:30:08.838280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.838193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2" podStartSLOduration=1.40401656 podStartE2EDuration="8.838180713s" totalImagesPullingTime="7.434164153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:08.83726902 +0000 UTC m=+756.959600953" watchObservedRunningTime="2026-09-11 21:30:08.838180713 +0000 UTC m=+756.960512634"
Sep 11 21:30:08.868554 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:08.868516    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92" podStartSLOduration=1.597482375 podStartE2EDuration="8.868486485s" totalImagesPullingTime="7.27100411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:08.866633692 +0000 UTC m=+756.988965612" watchObservedRunningTime="2026-09-11 21:30:08.868486485 +0000 UTC m=+756.990818448"
Sep 11 21:30:09.809650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.809606    2580 generic.go:356] "Generic (PLEG): container finished" podID="201f7648-1d93-4477-9cd1-5a3d67e39c5d" containerID="fd21884ef99ba1a9bc4018d5d5323c964c21d932dda076ed8c6ac76753a186c3" exitCode=0
Sep 11 21:30:09.810109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.809691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9" event={"ID":"201f7648-1d93-4477-9cd1-5a3d67e39c5d","Type":"ContainerDied","Data":"fd21884ef99ba1a9bc4018d5d5323c964c21d932dda076ed8c6ac76753a186c3"}
Sep 11 21:30:09.811350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.811322    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9f4a154-6bc2-4db5-af83-c9205d1accb0" containerID="2c1bd2400fed78efe7584d30829a770ceed75cb76a46a7e58fe5f0d9a0b82701" exitCode=0
Sep 11 21:30:09.811466 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.811408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92" event={"ID":"d9f4a154-6bc2-4db5-af83-c9205d1accb0","Type":"ContainerDied","Data":"2c1bd2400fed78efe7584d30829a770ceed75cb76a46a7e58fe5f0d9a0b82701"}
Sep 11 21:30:09.812975 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.812950    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15" containerID="26078ba125765f31b8eb7f3a94dfaa7e9f0506e7a85e9185bbde21bac4bb5c5b" exitCode=0
Sep 11 21:30:09.813106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:09.813016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2" event={"ID":"2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15","Type":"ContainerDied","Data":"26078ba125765f31b8eb7f3a94dfaa7e9f0506e7a85e9185bbde21bac4bb5c5b"}
Sep 11 21:30:10.978925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:10.978899    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"
Sep 11 21:30:11.015358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.015331    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"
Sep 11 21:30:11.036996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.036817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"
Sep 11 21:30:11.824242 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.824167    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerID="3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7" exitCode=0
Sep 11 21:30:11.824439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.824243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerDied","Data":"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"}
Sep 11 21:30:11.824686 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.824670    2580 scope.go:117] "RemoveContainer" containerID="3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"
Sep 11 21:30:11.825807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.825782    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"
Sep 11 21:30:11.825907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.825785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9" event={"ID":"201f7648-1d93-4477-9cd1-5a3d67e39c5d","Type":"ContainerDied","Data":"bfe097aaff656c5236521bd3c494cd81c9812f6588cc53f3f4b5759c9dcc6546"}
Sep 11 21:30:11.825970 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.825918    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe097aaff656c5236521bd3c494cd81c9812f6588cc53f3f4b5759c9dcc6546"
Sep 11 21:30:11.825970 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.825933    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:11.826067 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.826009    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:11.826067 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.826056    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:11.827158 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.827142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92"
Sep 11 21:30:11.827235 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.827160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-9tk92" event={"ID":"d9f4a154-6bc2-4db5-af83-c9205d1accb0","Type":"ContainerDied","Data":"703101bdebe4f4ffd6173586dda7a1c9b53637334fe74a6ee520827d7cd77289"}
Sep 11 21:30:11.827235 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.827196    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703101bdebe4f4ffd6173586dda7a1c9b53637334fe74a6ee520827d7cd77289"
Sep 11 21:30:11.833698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.830145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2" event={"ID":"2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15","Type":"ContainerDied","Data":"ade7b9b6a5680a2861621d38508475819957fbb3cb19ac7c1fd14b2c37505cf5"}
Sep 11 21:30:11.833698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.830180    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade7b9b6a5680a2861621d38508475819957fbb3cb19ac7c1fd14b2c37505cf5"
Sep 11 21:30:11.833698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:11.830229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"
Sep 11 21:30:12.179754 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.179733    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:30:12.297627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297714    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7b8\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token\") pod \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\" (UID: \"2d518ee7-e1fd-4be5-9602-c6e50b97226d\") "
Sep 11 21:30:12.297996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:12.297996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:12.298163 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.297986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:12.298163 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:30:12.298400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:12.298400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298400    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.298400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298419    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.298400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298430    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.298400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.298439    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.300052 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.300028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:12.300168 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.300076    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:30:12.300550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.300532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:30:12.300550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.300540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8" (OuterVolumeSpecName: "kube-api-access-5f7b8") pod "2d518ee7-e1fd-4be5-9602-c6e50b97226d" (UID: "2d518ee7-e1fd-4be5-9602-c6e50b97226d"). InnerVolumeSpecName "kube-api-access-5f7b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:30:12.399838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.399800    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.399838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.399836    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.399838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.399844    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.400037 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.399852    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d518ee7-e1fd-4be5-9602-c6e50b97226d-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.400037 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.399860    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7b8\" (UniqueName: \"kubernetes.io/projected/2d518ee7-e1fd-4be5-9602-c6e50b97226d-kube-api-access-5f7b8\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:12.507891 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.507853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:30:12.841120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.841075    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerID="86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573" exitCode=0
Sep 11 21:30:12.841329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.841188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerDied","Data":"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"}
Sep 11 21:30:12.841329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.841238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9" event={"ID":"2d518ee7-e1fd-4be5-9602-c6e50b97226d","Type":"ContainerDied","Data":"eb470c00c5f8ee9d3e4bfd19ebd4413358ac15ba6beb4d5f40cf164f6be46fe1"}
Sep 11 21:30:12.841329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.841270    2580 scope.go:117] "RemoveContainer" containerID="86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"
Sep 11 21:30:12.841519 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.841428    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"
Sep 11 21:30:12.846347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.846317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerStarted","Data":"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"}
Sep 11 21:30:12.860317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.860288    2580 scope.go:117] "RemoveContainer" containerID="52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"
Sep 11 21:30:12.870321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.870303    2580 scope.go:117] "RemoveContainer" containerID="86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"
Sep 11 21:30:12.870635 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:12.870614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573\": container with ID starting with 86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573 not found: ID does not exist" containerID="86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"
Sep 11 21:30:12.870690 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.870645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573"} err="failed to get container status \"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573\": rpc error: code = NotFound desc = could not find container \"86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573\": container with ID starting with 86b63c0dc078f1dcdcb21aa5bb85cd4120ab2819784b66adba01078e17eeb573 not found: ID does not exist"
Sep 11 21:30:12.870690 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.870662    2580 scope.go:117] "RemoveContainer" containerID="52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"
Sep 11 21:30:12.870923 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:12.870906    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734\": container with ID starting with 52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734 not found: ID does not exist" containerID="52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"
Sep 11 21:30:12.870962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.870929    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734"} err="failed to get container status \"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734\": rpc error: code = NotFound desc = could not find container \"52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734\": container with ID starting with 52d4847b20f2f5e550c62130db367cc0c021cbb71687a8b6612e37bbfe58f734 not found: ID does not exist"
Sep 11 21:30:12.888953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.888926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:30:12.898432 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:12.898412    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b37c5b2-openshift-default-66f9f46666-l68b9"]
Sep 11 21:30:13.281468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281428    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f4a154-6bc2-4db5-af83-c9205d1accb0" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281836    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f4a154-6bc2-4db5-af83-c9205d1accb0" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281847    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281852    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281865    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201f7648-1d93-4477-9cd1-5a3d67e39c5d" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281869    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="201f7648-1d93-4477-9cd1-5a3d67e39c5d" containerName="cleanup"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281875    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:13.281919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281880    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:13.282302 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="201f7648-1d93-4477-9cd1-5a3d67e39c5d" containerName="cleanup"
Sep 11 21:30:13.282302 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:13.282302 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281961    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:13.282302 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281967    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15" containerName="cleanup"
Sep 11 21:30:13.282302 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.281979    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f4a154-6bc2-4db5-af83-c9205d1accb0" containerName="cleanup"
Sep 11 21:30:13.285678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.285656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.288749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.288731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d3e394ce-serving-cert\""
Sep 11 21:30:13.304430 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.304408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:13.410367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.410332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.410367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.410370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.410640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.410400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.410640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.410515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnd4\" (UniqueName: \"kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.511317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.511285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnd4\" (UniqueName: \"kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.511468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.511379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.511468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.511405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.511468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.511449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.511614 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:13.511595    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-serving-cert: secret "maas-api-e2e-migrate-d3e394ce-serving-cert" not found
Sep 11 21:30:13.511684 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:13.511671    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls podName:5b1965b7-73b2-4c76-aca2-7cbd234e003a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:14.011647642 +0000 UTC m=+762.133979555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls") pod "maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a") : secret "maas-api-e2e-migrate-d3e394ce-serving-cert" not found
Sep 11 21:30:13.512123 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.512100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.513923 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.513907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.524164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.524144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnd4\" (UniqueName: \"kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:13.678250 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.678156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:30:13.678404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.678337    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:30:13.678404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:13.678380    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 11 21:30:14.017113 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:14.017084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:14.017275 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:14.017229    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-serving-cert: secret "maas-api-e2e-migrate-d3e394ce-serving-cert" not found
Sep 11 21:30:14.017313 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:14.017296    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls podName:5b1965b7-73b2-4c76-aca2-7cbd234e003a nodeName:}" failed. No retries permitted until 2026-09-11 21:30:15.017278378 +0000 UTC m=+763.139610276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls") pod "maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a") : secret "maas-api-e2e-migrate-d3e394ce-serving-cert" not found
Sep 11 21:30:14.383143 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:14.383065    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" path="/var/lib/kubelet/pods/2d518ee7-e1fd-4be5-9602-c6e50b97226d/volumes"
Sep 11 21:30:14.678596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:14.678491    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:30:14.678596 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:14.678571    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 11 21:30:15.035724 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.035676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:15.038999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.038948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:15.096884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.096852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:15.401999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.401975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:15.403525 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:30:15.403485    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1965b7_73b2_4c76_aca2_7cbd234e003a.slice/crio-5fb6a78f96481d237e58c940f3f816e8a5b66deac3001775d35a9bcbc5319ace WatchSource:0}: Error finding container 5fb6a78f96481d237e58c940f3f816e8a5b66deac3001775d35a9bcbc5319ace: Status 404 returned error can't find the container with id 5fb6a78f96481d237e58c940f3f816e8a5b66deac3001775d35a9bcbc5319ace
Sep 11 21:30:15.678662 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.678584    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:30:15.678662 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.678639    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 11 21:30:15.878450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:15.878426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" event={"ID":"5b1965b7-73b2-4c76-aca2-7cbd234e003a","Type":"ContainerStarted","Data":"5fb6a78f96481d237e58c940f3f816e8a5b66deac3001775d35a9bcbc5319ace"}
Sep 11 21:30:16.678779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:16.678748    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:30:16.679163 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:16.678801    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 11 21:30:16.885040 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:16.884998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" event={"ID":"5b1965b7-73b2-4c76-aca2-7cbd234e003a","Type":"ContainerStarted","Data":"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"}
Sep 11 21:30:16.885197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:16.885134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:16.916749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:16.916703    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" podStartSLOduration=3.528902574 podStartE2EDuration="3.916691911s" totalImagesPullingTime="387.789337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:16.913977342 +0000 UTC m=+765.036309261" watchObservedRunningTime="2026-09-11 21:30:16.916691911 +0000 UTC m=+765.039023831"
Sep 11 21:30:17.678397 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:17.678367    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 11 21:30:17.678585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:17.678422    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 11 21:30:18.682364 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:18.682334    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:30:18.682794 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:18.682743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:30:18.683334 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:18.683317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:30:22.895536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:22.895484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:29.956698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.956657    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerID="c686e86094ef33509b053d8d535ba3d5c6f1029b1d138ec32f922b06ff612857" exitCode=0
Sep 11 21:30:29.957095 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.956717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerDied","Data":"c686e86094ef33509b053d8d535ba3d5c6f1029b1d138ec32f922b06ff612857"}
Sep 11 21:30:29.957095 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.957062    2580 scope.go:117] "RemoveContainer" containerID="c686e86094ef33509b053d8d535ba3d5c6f1029b1d138ec32f922b06ff612857"
Sep 11 21:30:29.958119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.958092    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:29.958171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.958159    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:29.958208 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:29.958195    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:30:30.655246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:30.655194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:30:30.964338 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:30.964244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerStarted","Data":"d44846cf8e52b5c739ab43ae4341143483081fc5ca1ea4b293417f9c73846504"}
Sep 11 21:30:30.989344 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:30.989319    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:31.990512 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:31.990471    2580 patch_prober.go:28] interesting pod/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 21:30:31.990871 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:31.990553    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 21:30:32.990016 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:32.989980    2580 patch_prober.go:28] interesting pod/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 21:30:32.990180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:32.990030    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 21:30:33.990272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:33.990238    2580 patch_prober.go:28] interesting pod/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 21:30:33.990672 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:33.990289    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 21:30:34.989994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:34.989960    2580 patch_prober.go:28] interesting pod/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 21:30:34.990185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:34.990016    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 21:30:36.021608 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:36.021573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:36.022102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:36.022008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:36.022690 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:36.022666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:50.349113 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.349082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:50.349484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.349314    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" podUID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" containerName="maas-api" containerID="cri-o://ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115" gracePeriod=30
Sep 11 21:30:50.589049 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.589028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:50.679361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.679299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca\") pod \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") "
Sep 11 21:30:50.679477 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.679359    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhnd4\" (UniqueName: \"kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4\") pod \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") "
Sep 11 21:30:50.679477 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.679427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs\") pod \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") "
Sep 11 21:30:50.679589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.679482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") pod \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\" (UID: \"5b1965b7-73b2-4c76-aca2-7cbd234e003a\") "
Sep 11 21:30:50.679735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.679711    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b1965b7-73b2-4c76-aca2-7cbd234e003a" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:30:50.681757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.681733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b1965b7-73b2-4c76-aca2-7cbd234e003a" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:30:50.681840 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.681770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4" (OuterVolumeSpecName: "kube-api-access-nhnd4") pod "5b1965b7-73b2-4c76-aca2-7cbd234e003a" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a"). InnerVolumeSpecName "kube-api-access-nhnd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:30:50.681878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.681857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b1965b7-73b2-4c76-aca2-7cbd234e003a" (UID: "5b1965b7-73b2-4c76-aca2-7cbd234e003a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:30:50.780371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.780342    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:50.780371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.780367    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1965b7-73b2-4c76-aca2-7cbd234e003a-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:50.780371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.780375    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1965b7-73b2-4c76-aca2-7cbd234e003a-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:50.780595 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.780384    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhnd4\" (UniqueName: \"kubernetes.io/projected/5b1965b7-73b2-4c76-aca2-7cbd234e003a-kube-api-access-nhnd4\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:50.976420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.976345    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:50.976609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:50.976567    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" containerID="cri-o://d44846cf8e52b5c739ab43ae4341143483081fc5ca1ea4b293417f9c73846504" gracePeriod=30
Sep 11 21:30:51.024337 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.024311    2580 patch_prober.go:28] interesting pod/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:30:51.024465 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.024363    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:30:51.064170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.064141    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" containerID="ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115" exitCode=0
Sep 11 21:30:51.064299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.064176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" event={"ID":"5b1965b7-73b2-4c76-aca2-7cbd234e003a","Type":"ContainerDied","Data":"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"}
Sep 11 21:30:51.064299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.064190    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"
Sep 11 21:30:51.064299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.064208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd" event={"ID":"5b1965b7-73b2-4c76-aca2-7cbd234e003a","Type":"ContainerDied","Data":"5fb6a78f96481d237e58c940f3f816e8a5b66deac3001775d35a9bcbc5319ace"}
Sep 11 21:30:51.064299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.064230    2580 scope.go:117] "RemoveContainer" containerID="ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"
Sep 11 21:30:51.077547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.077408    2580 scope.go:117] "RemoveContainer" containerID="ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"
Sep 11 21:30:51.078035 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:51.077959    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115\": container with ID starting with ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115 not found: ID does not exist" containerID="ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"
Sep 11 21:30:51.078035 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.078008    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115"} err="failed to get container status \"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115\": rpc error: code = NotFound desc = could not find container \"ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115\": container with ID starting with ecf7660ceb95a95c30d08172acc1512238bca9776adcb0ee3ddf9937669dc115 not found: ID does not exist"
Sep 11 21:30:51.097212 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.097175    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:51.106833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.106800    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d3e394ce-6cfb977d58-hcjzd"]
Sep 11 21:30:51.701793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.701736    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:30:51.702172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.702121    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" containerName="maas-api"
Sep 11 21:30:51.702172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.702137    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" containerName="maas-api"
Sep 11 21:30:51.702172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.702162    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:51.702172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.702168    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d518ee7-e1fd-4be5-9602-c6e50b97226d" containerName="istio-proxy"
Sep 11 21:30:51.702351 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.702243    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" containerName="maas-api"
Sep 11 21:30:51.706488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.706473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.712094 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.712068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"default-dockercfg-vdx9q\""
Sep 11 21:30:51.712188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.712070    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"simulator-w0-db06d6-kserve-self-signed-certs\""
Sep 11 21:30:51.712188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.712111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"kube-root-ca.crt\""
Sep 11 21:30:51.712188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.712125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"openshift-service-ca.crt\""
Sep 11 21:30:51.718086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.718064    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:30:51.788698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.788851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.788851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.788851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.789015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.789015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.788926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.891883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.892472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.896525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.894152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.899103 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.898703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.900121 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.899970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:51.906831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:51.906810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p\") pod \"simulator-w0-db06d6-kserve-59698c6b5f-4zz4d\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") " pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:52.015294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:52.015261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:30:52.316575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:52.316554    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:30:52.318425 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:30:52.318396    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ed9750a_a332_42d8_99f9_5195a692470a.slice/crio-7385bc2335b93330e7b0202bc162c4ffc292700aef96e895545dd23e2c154227 WatchSource:0}: Error finding container 7385bc2335b93330e7b0202bc162c4ffc292700aef96e895545dd23e2c154227: Status 404 returned error can't find the container with id 7385bc2335b93330e7b0202bc162c4ffc292700aef96e895545dd23e2c154227
Sep 11 21:30:52.383723 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:52.383697    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1965b7-73b2-4c76-aca2-7cbd234e003a" path="/var/lib/kubelet/pods/5b1965b7-73b2-4c76-aca2-7cbd234e003a/volumes"
Sep 11 21:30:53.083206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:53.083176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" event={"ID":"9ed9750a-a332-42d8-99f9-5195a692470a","Type":"ContainerStarted","Data":"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"}
Sep 11 21:30:53.083206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:53.083211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" event={"ID":"9ed9750a-a332-42d8-99f9-5195a692470a","Type":"ContainerStarted","Data":"7385bc2335b93330e7b0202bc162c4ffc292700aef96e895545dd23e2c154227"}
Sep 11 21:30:53.108911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:53.108868    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" podStartSLOduration=2.108853215 podStartE2EDuration="2.108853215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:53.107086282 +0000 UTC m=+801.229418199" watchObservedRunningTime="2026-09-11 21:30:53.108853215 +0000 UTC m=+801.231185153"
Sep 11 21:30:56.100591 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.100559    2580 generic.go:356] "Generic (PLEG): container finished" podID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerID="d44846cf8e52b5c739ab43ae4341143483081fc5ca1ea4b293417f9c73846504" exitCode=0
Sep 11 21:30:56.100962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.100634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerDied","Data":"d44846cf8e52b5c739ab43ae4341143483081fc5ca1ea4b293417f9c73846504"}
Sep 11 21:30:56.100962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.100714    2580 scope.go:117] "RemoveContainer" containerID="c686e86094ef33509b053d8d535ba3d5c6f1029b1d138ec32f922b06ff612857"
Sep 11 21:30:56.221868 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.221845    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:56.338776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jxms\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.338776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338746    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.338953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.338953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338914    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.338953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.339117 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.338982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.339117 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.339117 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.339276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data\") pod \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\" (UID: \"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a\") "
Sep 11 21:30:56.339276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:56.339426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:56.339599 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339543    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.339673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:56.339673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339607    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.339673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:56.339788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.339522    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:30:56.341103 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.341077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:30:56.341573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.341550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:30:56.341680 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.341550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms" (OuterVolumeSpecName: "kube-api-access-5jxms") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "kube-api-access-5jxms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:30:56.341680 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.341578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" (UID: "9e2b07ec-61ec-42c3-a24f-8e37b571aa4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:30:56.441018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.440998    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxms\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-kube-api-access-5jxms\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441020    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441028    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441035    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441043    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441051    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.441149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.441059    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:30:56.843873 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:56.843838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:30:57.106844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:57.106762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7" event={"ID":"9e2b07ec-61ec-42c3-a24f-8e37b571aa4a","Type":"ContainerDied","Data":"5f8936dd903fbfe9dd62298ebe4ffa6f8dcadd1994519968676e18ef1e69aac9"}
Sep 11 21:30:57.106844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:57.106784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"
Sep 11 21:30:57.106844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:57.106818    2580 scope.go:117] "RemoveContainer" containerID="d44846cf8e52b5c739ab43ae4341143483081fc5ca1ea4b293417f9c73846504"
Sep 11 21:30:57.143800 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:57.143770    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:57.158609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:57.158587    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d3e394ce-gw-openshift-default-7f6dbc6ddd-6kml7"]
Sep 11 21:30:58.382601 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.382572    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" path="/var/lib/kubelet/pods/9e2b07ec-61ec-42c3-a24f-8e37b571aa4a/volumes"
Sep 11 21:30:58.923001 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.922972    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:30:58.923346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923333    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.923410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923349    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.923410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923372    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.923410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923378    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.923581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923437    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.923581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.923447    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2b07ec-61ec-42c3-a24f-8e37b571aa4a" containerName="istio-proxy"
Sep 11 21:30:58.931878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.931860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:58.936781 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.936759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0bde17e-serving-cert\""
Sep 11 21:30:58.947476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:58.947454    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:30:59.065438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.065399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.065610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.065536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.065610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.065562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.065610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.065589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhrr\" (UniqueName: \"kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.166661 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.166632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.166828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.166677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.166828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.166718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhrr\" (UniqueName: \"kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.166828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.166762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.166981 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:59.166872    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-serving-cert: secret "maas-api-e2e-ait-b0bde17e-serving-cert" not found
Sep 11 21:30:59.166981 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:30:59.166937    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls podName:c463cc4a-2a87-4577-a808-12a9e90333af nodeName:}" failed. No retries permitted until 2026-09-11 21:30:59.666916194 +0000 UTC m=+807.789248099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls") pod "maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" (UID: "c463cc4a-2a87-4577-a808-12a9e90333af") : secret "maas-api-e2e-ait-b0bde17e-serving-cert" not found
Sep 11 21:30:59.167398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.167380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.169304 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.169283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.183759 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.183703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhrr\" (UniqueName: \"kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.672280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.672250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.674607 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.674587    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") pod \"maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:30:59.844982 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:30:59.844952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:31:00.154695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:00.154669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:31:00.156947 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:31:00.156922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc463cc4a_2a87_4577_a808_12a9e90333af.slice/crio-51775578ee3fac84f4a6a947654e62f65628a9391fbefc5eee48bded32545459 WatchSource:0}: Error finding container 51775578ee3fac84f4a6a947654e62f65628a9391fbefc5eee48bded32545459: Status 404 returned error can't find the container with id 51775578ee3fac84f4a6a947654e62f65628a9391fbefc5eee48bded32545459
Sep 11 21:31:01.129232 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:01.129197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" event={"ID":"c463cc4a-2a87-4577-a808-12a9e90333af","Type":"ContainerStarted","Data":"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"}
Sep 11 21:31:01.129616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:01.129256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" event={"ID":"c463cc4a-2a87-4577-a808-12a9e90333af","Type":"ContainerStarted","Data":"51775578ee3fac84f4a6a947654e62f65628a9391fbefc5eee48bded32545459"}
Sep 11 21:31:01.129616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:01.129340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:31:01.168531 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:01.168473    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" podStartSLOduration=2.8202133849999997 podStartE2EDuration="3.168459823s" totalImagesPullingTime="348.246438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:01.167240072 +0000 UTC m=+809.289571991" watchObservedRunningTime="2026-09-11 21:31:01.168459823 +0000 UTC m=+809.290791741"
Sep 11 21:31:02.015578 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:02.015543    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:31:02.015742 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:02.015631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:31:02.027886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:02.027865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:31:02.145643 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:02.145615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:31:07.140652 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.140624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:31:07.298092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.298059    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:31:07.301629 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.301613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.304846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.304827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"default-dockercfg-p92xg\""
Sep 11 21:31:07.305435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.305416    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"openshift-service-ca.crt\""
Sep 11 21:31:07.305590 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.305569    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"simulator-w4-83ba87-kserve-self-signed-certs\""
Sep 11 21:31:07.305677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.305616    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"kube-root-ca.crt\""
Sep 11 21:31:07.314492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.314470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:31:07.440243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.440243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.440456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx746\" (UniqueName: \"kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.440456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.440456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.440456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.440425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541553 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541553 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541553 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541707 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx746\" (UniqueName: \"kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541707 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.541960 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.541912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.543878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.543853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.544001 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.543987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.551458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.551436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx746\" (UniqueName: \"kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746\") pod \"simulator-w4-83ba87-kserve-6b646c64c4-rlx25\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") " pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.610989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.610966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:07.891365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:07.891341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:31:07.893133 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:31:07.893105    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0214f63_8183_47d1_ab5c_c0ddf8f081b7.slice/crio-8275e5eb9a8d9f30bd39d2c33467e28bf7d05fc4134658d940b362b288231416 WatchSource:0}: Error finding container 8275e5eb9a8d9f30bd39d2c33467e28bf7d05fc4134658d940b362b288231416: Status 404 returned error can't find the container with id 8275e5eb9a8d9f30bd39d2c33467e28bf7d05fc4134658d940b362b288231416
Sep 11 21:31:08.183435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:08.183339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" event={"ID":"f0214f63-8183-47d1-ab5c-c0ddf8f081b7","Type":"ContainerStarted","Data":"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"}
Sep 11 21:31:08.184252 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:08.184223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" event={"ID":"f0214f63-8183-47d1-ab5c-c0ddf8f081b7","Type":"ContainerStarted","Data":"8275e5eb9a8d9f30bd39d2c33467e28bf7d05fc4134658d940b362b288231416"}
Sep 11 21:31:08.204495 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:08.204447    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" podStartSLOduration=1.204434126 podStartE2EDuration="1.204434126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:08.20389245 +0000 UTC m=+816.326224369" watchObservedRunningTime="2026-09-11 21:31:08.204434126 +0000 UTC m=+816.326766045"
Sep 11 21:31:13.177613 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.177583    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:31:13.181109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.181090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.183830 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.183798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1e9b41\"/\"default-dockercfg-8cpv8\""
Sep 11 21:31:13.184919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.184894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1e9b41\"/\"openshift-service-ca.crt\""
Sep 11 21:31:13.185009 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.184931    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1e9b41\"/\"kube-root-ca.crt\""
Sep 11 21:31:13.185009 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.184931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1e9b41\"/\"autores-model-1e9b41-kserve-self-signed-certs\""
Sep 11 21:31:13.193769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.193751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:31:13.291180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.291180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.291409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4zp\" (UniqueName: \"kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.291409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.291409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.291409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.291380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.391837 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.391809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.391848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.391898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.391934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.391993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4zp\" (UniqueName: \"kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392207 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.392017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.392259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.392285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.392404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.392384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.394308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.394290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.394579 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.394562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.401413 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.401390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4zp\" (UniqueName: \"kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp\") pod \"autores-model-1e9b41-kserve-589fb9559-cxqg9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") " pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.490376 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.490353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:13.785241 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:13.785219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:31:13.786967 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:31:13.786928    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3ce718_a689_4cd5_966a_dedbf9eb93e9.slice/crio-b3981ee7137f5a70d3c0f3d69c83bc54c2b092f6c6fbb772b70681279567cdd0 WatchSource:0}: Error finding container b3981ee7137f5a70d3c0f3d69c83bc54c2b092f6c6fbb772b70681279567cdd0: Status 404 returned error can't find the container with id b3981ee7137f5a70d3c0f3d69c83bc54c2b092f6c6fbb772b70681279567cdd0
Sep 11 21:31:14.226165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:14.226078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" event={"ID":"de3ce718-a689-4cd5-966a-dedbf9eb93e9","Type":"ContainerStarted","Data":"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"}
Sep 11 21:31:14.226165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:14.226132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" event={"ID":"de3ce718-a689-4cd5-966a-dedbf9eb93e9","Type":"ContainerStarted","Data":"b3981ee7137f5a70d3c0f3d69c83bc54c2b092f6c6fbb772b70681279567cdd0"}
Sep 11 21:31:14.247476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:14.247431    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" podStartSLOduration=1.247416818 podStartE2EDuration="1.247416818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:14.244201536 +0000 UTC m=+822.366533455" watchObservedRunningTime="2026-09-11 21:31:14.247416818 +0000 UTC m=+822.369748788"
Sep 11 21:31:17.611677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:17.611646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:17.612028 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:17.611696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:17.623457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:17.623431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:17.924321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:17.924248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:31:18.261077 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:18.261049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:31:23.490543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:23.490489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:23.490543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:23.490550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:23.502758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:23.502737    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:24.292369 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:24.292337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:31:32.615321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.615284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:31:32.620180 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.620162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.624062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.624043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1a1e3a\"/\"test-model-1a1e3a-kserve-self-signed-certs\""
Sep 11 21:31:32.624182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.624135    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1a1e3a\"/\"kube-root-ca.crt\""
Sep 11 21:31:32.624182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.624135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1a1e3a\"/\"default-dockercfg-rzdtd\""
Sep 11 21:31:32.624182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.624135    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1a1e3a\"/\"openshift-service-ca.crt\""
Sep 11 21:31:32.632980 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.632957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:31:32.664160 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.664160 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.664297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hxj\" (UniqueName: \"kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.664297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.664297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.664408 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.664338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.764945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.764912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765057 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.764961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765057 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765057 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765207 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hxj\" (UniqueName: \"kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765207 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765370 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.765538 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.765495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.767341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.767320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.767631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.767616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.773929 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.773910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hxj\" (UniqueName: \"kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj\") pod \"test-model-1a1e3a-kserve-754fbbdd4f-n66tk\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") " pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:32.930128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:32.930037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:33.215589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:33.215558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:31:33.216581 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:31:33.216558    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode444a08a_34b2_4cdb_9d6e_f0460e85e7a9.slice/crio-579799d93746b596780de677b8ddacd3e2adfe170d33a0cb62d6db51a9fe5b08 WatchSource:0}: Error finding container 579799d93746b596780de677b8ddacd3e2adfe170d33a0cb62d6db51a9fe5b08: Status 404 returned error can't find the container with id 579799d93746b596780de677b8ddacd3e2adfe170d33a0cb62d6db51a9fe5b08
Sep 11 21:31:33.327992 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:33.327956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" event={"ID":"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9","Type":"ContainerStarted","Data":"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"}
Sep 11 21:31:33.328134 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:33.328008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" event={"ID":"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9","Type":"ContainerStarted","Data":"579799d93746b596780de677b8ddacd3e2adfe170d33a0cb62d6db51a9fe5b08"}
Sep 11 21:31:33.350564 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:33.350488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" podStartSLOduration=1.350474429 podStartE2EDuration="1.350474429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:33.349224073 +0000 UTC m=+841.471555995" watchObservedRunningTime="2026-09-11 21:31:33.350474429 +0000 UTC m=+841.472806348"
Sep 11 21:31:36.412011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.411969    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:31:36.412420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.412196    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" podUID="c463cc4a-2a87-4577-a808-12a9e90333af" containerName="maas-api" containerID="cri-o://a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1" gracePeriod=30
Sep 11 21:31:36.667990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.667936    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:31:36.802600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.802567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") pod \"c463cc4a-2a87-4577-a808-12a9e90333af\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") "
Sep 11 21:31:36.802600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.802601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs\") pod \"c463cc4a-2a87-4577-a808-12a9e90333af\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") "
Sep 11 21:31:36.802783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.802657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca\") pod \"c463cc4a-2a87-4577-a808-12a9e90333af\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") "
Sep 11 21:31:36.802783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.802729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhrr\" (UniqueName: \"kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr\") pod \"c463cc4a-2a87-4577-a808-12a9e90333af\" (UID: \"c463cc4a-2a87-4577-a808-12a9e90333af\") "
Sep 11 21:31:36.803066 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.803043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c463cc4a-2a87-4577-a808-12a9e90333af" (UID: "c463cc4a-2a87-4577-a808-12a9e90333af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:31:36.804742 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.804711    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c463cc4a-2a87-4577-a808-12a9e90333af" (UID: "c463cc4a-2a87-4577-a808-12a9e90333af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:31:36.804835 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.804761    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c463cc4a-2a87-4577-a808-12a9e90333af" (UID: "c463cc4a-2a87-4577-a808-12a9e90333af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:31:36.804893 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.804869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr" (OuterVolumeSpecName: "kube-api-access-vhhrr") pod "c463cc4a-2a87-4577-a808-12a9e90333af" (UID: "c463cc4a-2a87-4577-a808-12a9e90333af"). InnerVolumeSpecName "kube-api-access-vhhrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:31:36.904251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.904220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhrr\" (UniqueName: \"kubernetes.io/projected/c463cc4a-2a87-4577-a808-12a9e90333af-kube-api-access-vhhrr\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:36.904251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.904248    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:36.904251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.904257    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c463cc4a-2a87-4577-a808-12a9e90333af-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:36.904468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:36.904265    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463cc4a-2a87-4577-a808-12a9e90333af-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:37.353273 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.353235    2580 generic.go:356] "Generic (PLEG): container finished" podID="c463cc4a-2a87-4577-a808-12a9e90333af" containerID="a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1" exitCode=0
Sep 11 21:31:37.353484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.353281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"
Sep 11 21:31:37.353484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.353319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" event={"ID":"c463cc4a-2a87-4577-a808-12a9e90333af","Type":"ContainerDied","Data":"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"}
Sep 11 21:31:37.353484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.353373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj" event={"ID":"c463cc4a-2a87-4577-a808-12a9e90333af","Type":"ContainerDied","Data":"51775578ee3fac84f4a6a947654e62f65628a9391fbefc5eee48bded32545459"}
Sep 11 21:31:37.353484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.353396    2580 scope.go:117] "RemoveContainer" containerID="a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"
Sep 11 21:31:37.362717 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.362698    2580 scope.go:117] "RemoveContainer" containerID="a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"
Sep 11 21:31:37.362981 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:31:37.362963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1\": container with ID starting with a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1 not found: ID does not exist" containerID="a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"
Sep 11 21:31:37.363024 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.362990    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1"} err="failed to get container status \"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1\": rpc error: code = NotFound desc = could not find container \"a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1\": container with ID starting with a553499d4d0257bbb0a1d7a0df81bb3051d969dc894c2734ac0c114757db64a1 not found: ID does not exist"
Sep 11 21:31:37.376758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.376733    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:31:37.381012 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:37.380991    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0bde17e-676b7cc67-kh6pj"]
Sep 11 21:31:38.383275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:38.383244    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c463cc4a-2a87-4577-a808-12a9e90333af" path="/var/lib/kubelet/pods/c463cc4a-2a87-4577-a808-12a9e90333af/volumes"
Sep 11 21:31:39.121763 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:39.116151    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:31:42.930893 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:42.930862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:42.931272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:42.930917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:42.943924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:42.943893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:43.396972 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:43.396944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:31:59.204229 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.204195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:31:59.204734 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.204396    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" podUID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" containerName="maas-api" containerID="cri-o://2b7a46ceaa57be68f1277c7ec46e35b9a5a548271ea99b01d26952d79e4a4ebd" gracePeriod=30
Sep 11 21:31:59.249978 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.249952    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"]
Sep 11 21:31:59.252531 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.252493    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1e9b41-29819370-jlnx9"]
Sep 11 21:31:59.465836 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.465772    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" containerID="2b7a46ceaa57be68f1277c7ec46e35b9a5a548271ea99b01d26952d79e4a4ebd" exitCode=0
Sep 11 21:31:59.465836 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.465820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" event={"ID":"7ff3303a-fb03-4188-a6d2-be32fdf4305a","Type":"ContainerDied","Data":"2b7a46ceaa57be68f1277c7ec46e35b9a5a548271ea99b01d26952d79e4a4ebd"}
Sep 11 21:31:59.466007 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.465872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r" event={"ID":"7ff3303a-fb03-4188-a6d2-be32fdf4305a","Type":"ContainerDied","Data":"06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e"}
Sep 11 21:31:59.466007 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.465892    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d78721e760e6562f4ca0468e2aed01c1e3aff106bf15ddab522ae7bba4695e"
Sep 11 21:31:59.468474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.468452    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:31:59.613845 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.613811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca\") pod \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") "
Sep 11 21:31:59.614231 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.613948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccnjb\" (UniqueName: \"kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb\") pod \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") "
Sep 11 21:31:59.614231 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.613983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls\") pod \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") "
Sep 11 21:31:59.614231 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.614039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs\") pod \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\" (UID: \"7ff3303a-fb03-4188-a6d2-be32fdf4305a\") "
Sep 11 21:31:59.614231 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.614217    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ff3303a-fb03-4188-a6d2-be32fdf4305a" (UID: "7ff3303a-fb03-4188-a6d2-be32fdf4305a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:31:59.614976 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.614372    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff3303a-fb03-4188-a6d2-be32fdf4305a-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:59.618168 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.617168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ff3303a-fb03-4188-a6d2-be32fdf4305a" (UID: "7ff3303a-fb03-4188-a6d2-be32fdf4305a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:31:59.618533 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.618482    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb" (OuterVolumeSpecName: "kube-api-access-ccnjb") pod "7ff3303a-fb03-4188-a6d2-be32fdf4305a" (UID: "7ff3303a-fb03-4188-a6d2-be32fdf4305a"). InnerVolumeSpecName "kube-api-access-ccnjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:31:59.618982 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.618959    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ff3303a-fb03-4188-a6d2-be32fdf4305a" (UID: "7ff3303a-fb03-4188-a6d2-be32fdf4305a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:31:59.715459 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.715427    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccnjb\" (UniqueName: \"kubernetes.io/projected/7ff3303a-fb03-4188-a6d2-be32fdf4305a-kube-api-access-ccnjb\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:59.715459 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.715454    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:31:59.715459 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:31:59.715463    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff3303a-fb03-4188-a6d2-be32fdf4305a-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:00.383677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:00.383643    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201f7648-1d93-4477-9cd1-5a3d67e39c5d" path="/var/lib/kubelet/pods/201f7648-1d93-4477-9cd1-5a3d67e39c5d/volumes"
Sep 11 21:32:00.471170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:00.471140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"
Sep 11 21:32:00.506079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:00.506052    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:32:00.520410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:00.520384    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1e9b41-694fbf78cd-trq5r"]
Sep 11 21:32:01.186824 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.186789    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"]
Sep 11 21:32:01.187181 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187168    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" containerName="maas-api"
Sep 11 21:32:01.187228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187183    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" containerName="maas-api"
Sep 11 21:32:01.187228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c463cc4a-2a87-4577-a808-12a9e90333af" containerName="maas-api"
Sep 11 21:32:01.187228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c463cc4a-2a87-4577-a808-12a9e90333af" containerName="maas-api"
Sep 11 21:32:01.187322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187252    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" containerName="maas-api"
Sep 11 21:32:01.187322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.187259    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c463cc4a-2a87-4577-a808-12a9e90333af" containerName="maas-api"
Sep 11 21:32:01.190261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.190247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:01.195198 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.195178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nhbxk\""
Sep 11 21:32:01.195971 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.195944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:32:01.206458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.206436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"]
Sep 11 21:32:01.331176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.331147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh\" (UID: \"4db8d769-195d-4913-a1a9-0bd4f6cc1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:01.432167 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.432137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh\" (UID: \"4db8d769-195d-4913-a1a9-0bd4f6cc1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:01.432759 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.432742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh\" (UID: \"4db8d769-195d-4913-a1a9-0bd4f6cc1e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:01.499672 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.499651    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:01.783120 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:01.783095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"]
Sep 11 21:32:01.785089 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:32:01.785063    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db8d769_195d_4913_a1a9_0bd4f6cc1e43.slice/crio-f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9 WatchSource:0}: Error finding container f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9: Status 404 returned error can't find the container with id f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9
Sep 11 21:32:02.383482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:02.383402    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff3303a-fb03-4188-a6d2-be32fdf4305a" path="/var/lib/kubelet/pods/7ff3303a-fb03-4188-a6d2-be32fdf4305a/volumes"
Sep 11 21:32:02.482279 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:02.482247    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh" event={"ID":"4db8d769-195d-4913-a1a9-0bd4f6cc1e43","Type":"ContainerStarted","Data":"457cb73b0081f719bb15828898c85a669bd8b53190588d75820197be6e08a750"}
Sep 11 21:32:02.482616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:02.482295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh" event={"ID":"4db8d769-195d-4913-a1a9-0bd4f6cc1e43","Type":"ContainerStarted","Data":"f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9"}
Sep 11 21:32:02.511510 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:02.511451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh" podStartSLOduration=1.511436465 podStartE2EDuration="1.511436465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:02.510476533 +0000 UTC m=+870.632808453" watchObservedRunningTime="2026-09-11 21:32:02.511436465 +0000 UTC m=+870.633768384"
Sep 11 21:32:03.488666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:03.488631    2580 generic.go:356] "Generic (PLEG): container finished" podID="4db8d769-195d-4913-a1a9-0bd4f6cc1e43" containerID="457cb73b0081f719bb15828898c85a669bd8b53190588d75820197be6e08a750" exitCode=0
Sep 11 21:32:03.489025 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:03.488711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh" event={"ID":"4db8d769-195d-4913-a1a9-0bd4f6cc1e43","Type":"ContainerDied","Data":"457cb73b0081f719bb15828898c85a669bd8b53190588d75820197be6e08a750"}
Sep 11 21:32:04.620740 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:04.620714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:04.762160 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:04.762085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca\") pod \"4db8d769-195d-4913-a1a9-0bd4f6cc1e43\" (UID: \"4db8d769-195d-4913-a1a9-0bd4f6cc1e43\") "
Sep 11 21:32:04.762384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:04.762365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4db8d769-195d-4913-a1a9-0bd4f6cc1e43" (UID: "4db8d769-195d-4913-a1a9-0bd4f6cc1e43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:04.848590 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:04.848565    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:32:04.863311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:04.863285    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db8d769-195d-4913-a1a9-0bd4f6cc1e43-maas-api-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:05.500272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:05.500242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"
Sep 11 21:32:05.500434 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:05.500276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh" event={"ID":"4db8d769-195d-4913-a1a9-0bd4f6cc1e43","Type":"ContainerDied","Data":"f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9"}
Sep 11 21:32:05.500434 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:05.500324    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1843be6efbb173ce8a82e8a268261a1400f6e86b1d594b590a3dd6439698bd9"
Sep 11 21:32:06.091688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:06.091659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:32:12.428807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:12.428768    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"]
Sep 11 21:32:12.433887 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:12.433861    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0cd86cdc-7rplh"]
Sep 11 21:32:14.383382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:14.383342    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db8d769-195d-4913-a1a9-0bd4f6cc1e43" path="/var/lib/kubelet/pods/4db8d769-195d-4913-a1a9-0bd4f6cc1e43/volumes"
Sep 11 21:32:19.867672 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:19.867631    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" podUID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" containerName="main" containerID="cri-o://ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df" gracePeriod=45
Sep 11 21:32:20.119299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.119243    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:32:20.198526 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg4zp\" (UniqueName: \"kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp\") pod \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\" (UID: \"de3ce718-a689-4cd5-966a-dedbf9eb93e9\") "
Sep 11 21:32:20.198834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:20.199000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home" (OuterVolumeSpecName: "home") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:20.199000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.198917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:20.199109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.199052    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.199109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.199066    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.199109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.199075    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.200701 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.200681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm" (OuterVolumeSpecName: "dshm") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:20.200856 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.200831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:32:20.200910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.200870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp" (OuterVolumeSpecName: "kube-api-access-tg4zp") pod "de3ce718-a689-4cd5-966a-dedbf9eb93e9" (UID: "de3ce718-a689-4cd5-966a-dedbf9eb93e9"). InnerVolumeSpecName "kube-api-access-tg4zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:20.299928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.299898    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de3ce718-a689-4cd5-966a-dedbf9eb93e9-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.299928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.299921    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de3ce718-a689-4cd5-966a-dedbf9eb93e9-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.299928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.299932    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4zp\" (UniqueName: \"kubernetes.io/projected/de3ce718-a689-4cd5-966a-dedbf9eb93e9-kube-api-access-tg4zp\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:20.583608 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.583576    2580 generic.go:356] "Generic (PLEG): container finished" podID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" containerID="ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df" exitCode=0
Sep 11 21:32:20.583746 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.583624    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"
Sep 11 21:32:20.583746 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.583658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" event={"ID":"de3ce718-a689-4cd5-966a-dedbf9eb93e9","Type":"ContainerDied","Data":"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"}
Sep 11 21:32:20.583746 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.583699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9" event={"ID":"de3ce718-a689-4cd5-966a-dedbf9eb93e9","Type":"ContainerDied","Data":"b3981ee7137f5a70d3c0f3d69c83bc54c2b092f6c6fbb772b70681279567cdd0"}
Sep 11 21:32:20.583746 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.583722    2580 scope.go:117] "RemoveContainer" containerID="ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"
Sep 11 21:32:20.591980 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.591966    2580 scope.go:117] "RemoveContainer" containerID="ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"
Sep 11 21:32:20.592274 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:32:20.592256    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df\": container with ID starting with ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df not found: ID does not exist" containerID="ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"
Sep 11 21:32:20.592339 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.592285    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df"} err="failed to get container status \"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df\": rpc error: code = NotFound desc = could not find container \"ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df\": container with ID starting with ead1b166542b33a75624e805e5882c1e0071bef13ee2c69d74a9a86d469f75df not found: ID does not exist"
Sep 11 21:32:20.600271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.600245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:32:20.603802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.603780    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1e9b41/autores-model-1e9b41-kserve-589fb9559-cxqg9"]
Sep 11 21:32:20.847488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:20.847456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:32:22.384144 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:22.384110    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" path="/var/lib/kubelet/pods/de3ce718-a689-4cd5-966a-dedbf9eb93e9/volumes"
Sep 11 21:32:41.109569 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109542    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:32:41.109941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109858    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" containerName="main"
Sep 11 21:32:41.109941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109869    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" containerName="main"
Sep 11 21:32:41.109941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109878    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db8d769-195d-4913-a1a9-0bd4f6cc1e43" containerName="revoke-keys"
Sep 11 21:32:41.109941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db8d769-195d-4913-a1a9-0bd4f6cc1e43" containerName="revoke-keys"
Sep 11 21:32:41.110108 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109949    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db8d769-195d-4913-a1a9-0bd4f6cc1e43" containerName="revoke-keys"
Sep 11 21:32:41.110108 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.109958    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3ce718-a689-4cd5-966a-dedbf9eb93e9" containerName="main"
Sep 11 21:32:41.114132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.114115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.118362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.118345    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"premium-w0-db06d6-kserve-self-signed-certs\""
Sep 11 21:32:41.128005 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.127983    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:32:41.262724 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.262883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.262883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.262883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.262883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.263039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.262895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck5k\" (UniqueName: \"kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck5k\" (UniqueName: \"kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364714 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.364778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.364754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.366637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.366609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.366811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.366794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.376831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.376799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck5k\" (UniqueName: \"kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k\") pod \"premium-w0-db06d6-kserve-7fd6d46578-kwxq8\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") " pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.422903 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.422883    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:41.697919 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:41.697897    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:32:41.699972 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:32:41.699944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772ef7ba_034e_446d_ac99_b0f4f5190d7e.slice/crio-6ace35eaddfe7df2a56ce1129928b35d8eb6b1d012feff88ae4af8c49d60b9a0 WatchSource:0}: Error finding container 6ace35eaddfe7df2a56ce1129928b35d8eb6b1d012feff88ae4af8c49d60b9a0: Status 404 returned error can't find the container with id 6ace35eaddfe7df2a56ce1129928b35d8eb6b1d012feff88ae4af8c49d60b9a0
Sep 11 21:32:42.670945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:42.670913    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:32:42.671378 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:42.671133    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy" containerID="cri-o://211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a" gracePeriod=30
Sep 11 21:32:42.696833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:42.696800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" event={"ID":"772ef7ba-034e-446d-ac99-b0f4f5190d7e","Type":"ContainerStarted","Data":"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"}
Sep 11 21:32:42.696956 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:42.696838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" event={"ID":"772ef7ba-034e-446d-ac99-b0f4f5190d7e","Type":"ContainerStarted","Data":"6ace35eaddfe7df2a56ce1129928b35d8eb6b1d012feff88ae4af8c49d60b9a0"}
Sep 11 21:32:42.716321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:42.716274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" podStartSLOduration=1.716259301 podStartE2EDuration="1.716259301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:42.71439133 +0000 UTC m=+910.836723264" watchObservedRunningTime="2026-09-11 21:32:42.716259301 +0000 UTC m=+910.838591221"
Sep 11 21:32:47.916443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:47.916417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:32:48.028751 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.028910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028784    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.028910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.028910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpknn\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.028963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029001    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket\") pod \"5d0c74ed-33cb-4d97-929b-b417861f5f03\" (UID: \"5d0c74ed-33cb-4d97-929b-b417861f5f03\") "
Sep 11 21:32:48.029326 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:48.029379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029368    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0c74ed-33cb-4d97-929b-b417861f5f03-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.029464 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:48.029766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:48.029766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:48.029935 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.029910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:48.031666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.031636    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:32:48.032200 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.032168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn" (OuterVolumeSpecName: "kube-api-access-cpknn") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "kube-api-access-cpknn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:48.032677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.032653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:48.033311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.033283    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d0c74ed-33cb-4d97-929b-b417861f5f03" (UID: "5d0c74ed-33cb-4d97-929b-b417861f5f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:48.098266 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.098240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:32:48.130413 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130386    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130413 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130412    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130420    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpknn\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-kube-api-access-cpknn\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130430    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130440    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130452    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130461    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.130633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.130468    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0c74ed-33cb-4d97-929b-b417861f5f03-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:32:48.732409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.732318    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerID="211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a" exitCode=0
Sep 11 21:32:48.732587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.732423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerDied","Data":"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"}
Sep 11 21:32:48.732587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.732492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2" event={"ID":"5d0c74ed-33cb-4d97-929b-b417861f5f03","Type":"ContainerDied","Data":"3d9d9baeee00713f85c2fdef072a82a94643dcf6d82d7c96969fddd044705ad6"}
Sep 11 21:32:48.732587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.732542    2580 scope.go:117] "RemoveContainer" containerID="211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"
Sep 11 21:32:48.732587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.732425    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"
Sep 11 21:32:48.749318 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.747936    2580 scope.go:117] "RemoveContainer" containerID="b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"
Sep 11 21:32:48.759873 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.759829    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:32:48.771547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.770120    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1e9b41-openshift-default-65f7b545f8-9lxc2"]
Sep 11 21:32:48.771547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.771032    2580 scope.go:117] "RemoveContainer" containerID="211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"
Sep 11 21:32:48.771547 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:32:48.771389    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a\": container with ID starting with 211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a not found: ID does not exist" containerID="211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"
Sep 11 21:32:48.771547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.771472    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a"} err="failed to get container status \"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a\": rpc error: code = NotFound desc = could not find container \"211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a\": container with ID starting with 211cbdace48802bee5535905d148ce015e3f36e663bb3b380b63af535b57724a not found: ID does not exist"
Sep 11 21:32:48.771928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.771910    2580 scope.go:117] "RemoveContainer" containerID="b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"
Sep 11 21:32:48.772392 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:32:48.772341    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189\": container with ID starting with b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189 not found: ID does not exist" containerID="b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"
Sep 11 21:32:48.772392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:48.772368    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189"} err="failed to get container status \"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189\": rpc error: code = NotFound desc = could not find container \"b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189\": container with ID starting with b28e11dc94c6a3f1f57cd3cef2b3458de878d945c54f048cdce4a5fb4e708189 not found: ID does not exist"
Sep 11 21:32:49.875161 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875130    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:32:49.875619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:32:49.875619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875478    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:32:49.875619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875485    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:32:49.875619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875490    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:32:49.875619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.875567    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:32:49.880176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.880154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:49.883017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.882855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c0162b-serving-cert\""
Sep 11 21:32:49.883017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.882907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:32:49.883017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.882957    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czlrt\""
Sep 11 21:32:49.883729 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.883714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:32:49.883831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.883731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:32:49.888584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:49.888562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:32:50.051567 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.051530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5tj\" (UniqueName: \"kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.051749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.051645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.051749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.051712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.051749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.051736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.152192 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.152123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.152192 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.152175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.152378 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.152197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.152378 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.152256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5tj\" (UniqueName: \"kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.152378 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:32:50.152271    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-serving-cert: secret "maas-api-e2e-explicit-c0162b-serving-cert" not found
Sep 11 21:32:50.152378 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:32:50.152339    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls podName:a46c4527-754c-4c7d-b4b2-d5fe939b85cd nodeName:}" failed. No retries permitted until 2026-09-11 21:32:50.652321985 +0000 UTC m=+918.774653887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls") pod "maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" (UID: "a46c4527-754c-4c7d-b4b2-d5fe939b85cd") : secret "maas-api-e2e-explicit-c0162b-serving-cert" not found
Sep 11 21:32:50.153036 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.153012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.154730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.154712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.163271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.163241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5tj\" (UniqueName: \"kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.383698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.383666    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" path="/var/lib/kubelet/pods/5d0c74ed-33cb-4d97-929b-b417861f5f03/volumes"
Sep 11 21:32:50.656329 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.656296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.658889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.658863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") pod \"maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:50.792566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:50.792537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:51.072661 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.072622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:32:51.074562 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:32:51.074542    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c4527_754c_4c7d_b4b2_d5fe939b85cd.slice/crio-02f5b596300391574748d399b6f49259b33ddc6b15c1990fca8e6710a0fd9109 WatchSource:0}: Error finding container 02f5b596300391574748d399b6f49259b33ddc6b15c1990fca8e6710a0fd9109: Status 404 returned error can't find the container with id 02f5b596300391574748d399b6f49259b33ddc6b15c1990fca8e6710a0fd9109
Sep 11 21:32:51.075887 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.075870    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:32:51.423264 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.423192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:51.423441 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.423424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:51.436594 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.436569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:51.750830 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.750785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" event={"ID":"a46c4527-754c-4c7d-b4b2-d5fe939b85cd","Type":"ContainerStarted","Data":"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"}
Sep 11 21:32:51.750981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.750889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:32:51.750981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.750916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" event={"ID":"a46c4527-754c-4c7d-b4b2-d5fe939b85cd","Type":"ContainerStarted","Data":"02f5b596300391574748d399b6f49259b33ddc6b15c1990fca8e6710a0fd9109"}
Sep 11 21:32:51.761844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.761824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:32:51.770644 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:51.770580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" podStartSLOduration=2.39404253 podStartE2EDuration="2.770564961s" totalImagesPullingTime="376.522431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:51.769809572 +0000 UTC m=+919.892141492" watchObservedRunningTime="2026-09-11 21:32:51.770564961 +0000 UTC m=+919.892896879"
Sep 11 21:32:57.762131 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:32:57.762099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:33:07.594929 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:07.594901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:33:10.190495 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.190464    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:33:10.191079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.191057    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d0c74ed-33cb-4d97-929b-b417861f5f03" containerName="istio-proxy"
Sep 11 21:33:10.194309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.194291    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.196686 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.196663    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"premium-w4-83ba87-kserve-self-signed-certs\""
Sep 11 21:33:10.204420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.204397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:33:10.218928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.218906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.219022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.218941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.219022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.218988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.219122 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.219022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.219122 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.219057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.219191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.219120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqxk\" (UniqueName: \"kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.319775 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.319746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.319928 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.319793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.320188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.320163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.320366 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.320349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.320484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.320471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.320616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.320600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqxk\" (UniqueName: \"kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.321381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.320494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.321600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.321579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.321756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.321640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.323925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.323838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.324422 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.324400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.329749 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.329729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqxk\" (UniqueName: \"kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk\") pod \"premium-w4-83ba87-kserve-66c596bf4c-h74t9\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") " pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.505397 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.505365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:10.776112 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.776084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:33:10.778033 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:33:10.778008    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedd3c8af_36ff_40fe_be3c_13aa48298e39.slice/crio-859cccb9fd6fe88fef43db06f6c59d1f45a9f33daca13cf79505cf6a63003bc3 WatchSource:0}: Error finding container 859cccb9fd6fe88fef43db06f6c59d1f45a9f33daca13cf79505cf6a63003bc3: Status 404 returned error can't find the container with id 859cccb9fd6fe88fef43db06f6c59d1f45a9f33daca13cf79505cf6a63003bc3
Sep 11 21:33:10.847734 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.847704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" event={"ID":"edd3c8af-36ff-40fe-be3c-13aa48298e39","Type":"ContainerStarted","Data":"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"}
Sep 11 21:33:10.847844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.847747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" event={"ID":"edd3c8af-36ff-40fe-be3c-13aa48298e39","Type":"ContainerStarted","Data":"859cccb9fd6fe88fef43db06f6c59d1f45a9f33daca13cf79505cf6a63003bc3"}
Sep 11 21:33:10.864750 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:10.864709    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" podStartSLOduration=0.8646932 podStartE2EDuration="864.6932ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:33:10.863772221 +0000 UTC m=+938.986104158" watchObservedRunningTime="2026-09-11 21:33:10.8646932 +0000 UTC m=+938.987025118"
Sep 11 21:33:20.505942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:20.505908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:20.505942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:20.505949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:20.518331 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:20.518300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:20.917568 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:20.917481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:33:23.985641 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:23.985609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:33:24.787684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.787645    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:33:24.792342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.792324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.804604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.804583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02c60f\"/\"kube-root-ca.crt\""
Sep 11 21:33:24.804828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.804812    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02c60f\"/\"test-model-02c60f-kserve-self-signed-certs\""
Sep 11 21:33:24.805620 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.805606    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02c60f\"/\"default-dockercfg-6fvx5\""
Sep 11 21:33:24.805674 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.805658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02c60f\"/\"openshift-service-ca.crt\""
Sep 11 21:33:24.810756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.810736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:33:24.859523 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.859635 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.859635 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.859635 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.859780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.859780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.859730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268r9\" (UniqueName: \"kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960110 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960435 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960258    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268r9\" (UniqueName: \"kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960646 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.960646 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.960614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.962639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.962615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.962892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.962875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:24.971639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:24.971621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-268r9\" (UniqueName: \"kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9\") pod \"test-model-02c60f-kserve-6b7d844587-sgqvq\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") " pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:25.101624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:25.101540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:25.373223 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:25.373194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:33:25.374992 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:33:25.374962    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95051592_ffd6_4a5e_bf2f_a9e3ae28a59e.slice/crio-1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7 WatchSource:0}: Error finding container 1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7: Status 404 returned error can't find the container with id 1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7
Sep 11 21:33:25.935276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:25.935241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" event={"ID":"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e","Type":"ContainerStarted","Data":"c314128c2361f2550cfc25b0c5616a02f24d2f07ba84352475bcb59a32c37e4f"}
Sep 11 21:33:25.935276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:25.935282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" event={"ID":"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e","Type":"ContainerStarted","Data":"1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7"}
Sep 11 21:33:25.952942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:25.952896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" podStartSLOduration=1.9528819849999999 podStartE2EDuration="1.952881985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:33:25.950905311 +0000 UTC m=+954.073237231" watchObservedRunningTime="2026-09-11 21:33:25.952881985 +0000 UTC m=+954.075213901"
Sep 11 21:33:35.101870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:35.101835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:35.102220 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:35.101916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:35.114282 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:35.114263    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:35.998451 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:35.998427    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:33:45.285834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:45.285795    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:33:46.593373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.593345    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:33:46.597053 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.597033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.600565 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.600545    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c0162b\"/\"openshift-service-ca.crt\""
Sep 11 21:33:46.600745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.600725    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c0162b\"/\"default-dockercfg-92l7w\""
Sep 11 21:33:46.600815 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.600725    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c0162b\"/\"explicit-model-c0162b-kserve-self-signed-certs\""
Sep 11 21:33:46.600815 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.600725    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c0162b\"/\"kube-root-ca.crt\""
Sep 11 21:33:46.607251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.607232    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:33:46.755479 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.755655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt87\" (UniqueName: \"kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.755655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.755741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.755741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.755817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.755809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt87\" (UniqueName: \"kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857364 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857588 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.857687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.857649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.859838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.859812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.859942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.859926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.868822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.868792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt87\" (UniqueName: \"kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87\") pod \"explicit-model-c0162b-kserve-6957b894cc-6fc8s\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") " pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:46.906175 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:46.906148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:47.186144 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:47.186117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:33:47.188262 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:33:47.188239    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13110bb9_bfc2_4b6b_ad76_0ced2bdb69f7.slice/crio-b414a49d2e3db9d1a04abcd87791bc6e690648db4a2a9507793e38ae440b5099 WatchSource:0}: Error finding container b414a49d2e3db9d1a04abcd87791bc6e690648db4a2a9507793e38ae440b5099: Status 404 returned error can't find the container with id b414a49d2e3db9d1a04abcd87791bc6e690648db4a2a9507793e38ae440b5099
Sep 11 21:33:48.055401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:48.055366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" event={"ID":"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7","Type":"ContainerStarted","Data":"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"}
Sep 11 21:33:48.055808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:48.055407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" event={"ID":"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7","Type":"ContainerStarted","Data":"b414a49d2e3db9d1a04abcd87791bc6e690648db4a2a9507793e38ae440b5099"}
Sep 11 21:33:48.071892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:48.071843    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" podStartSLOduration=2.07182993 podStartE2EDuration="2.07182993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:33:48.071631993 +0000 UTC m=+976.193963913" watchObservedRunningTime="2026-09-11 21:33:48.07182993 +0000 UTC m=+976.194161848"
Sep 11 21:33:56.906756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:56.906662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:56.907121 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:56.906751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:56.919131 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:56.919102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:57.114828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:57.114801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:33:59.869962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:33:59.869930    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:34:10.864334 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:10.864301    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:34:10.864815 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:10.864562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" podUID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" containerName="maas-api" containerID="cri-o://efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7" gracePeriod=30
Sep 11 21:34:11.113040 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.113018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:34:11.180341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.180270    2580 generic.go:356] "Generic (PLEG): container finished" podID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" containerID="efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7" exitCode=0
Sep 11 21:34:11.180341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.180316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"
Sep 11 21:34:11.180341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.180334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" event={"ID":"a46c4527-754c-4c7d-b4b2-d5fe939b85cd","Type":"ContainerDied","Data":"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"}
Sep 11 21:34:11.180602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.180378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc" event={"ID":"a46c4527-754c-4c7d-b4b2-d5fe939b85cd","Type":"ContainerDied","Data":"02f5b596300391574748d399b6f49259b33ddc6b15c1990fca8e6710a0fd9109"}
Sep 11 21:34:11.180602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.180403    2580 scope.go:117] "RemoveContainer" containerID="efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"
Sep 11 21:34:11.182728 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.182708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5tj\" (UniqueName: \"kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj\") pod \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") "
Sep 11 21:34:11.182835 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.182800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") pod \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") "
Sep 11 21:34:11.182904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.182856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs\") pod \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") "
Sep 11 21:34:11.182904 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.182885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca\") pod \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\" (UID: \"a46c4527-754c-4c7d-b4b2-d5fe939b85cd\") "
Sep 11 21:34:11.183311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.183276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a46c4527-754c-4c7d-b4b2-d5fe939b85cd" (UID: "a46c4527-754c-4c7d-b4b2-d5fe939b85cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:34:11.185257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.185227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a46c4527-754c-4c7d-b4b2-d5fe939b85cd" (UID: "a46c4527-754c-4c7d-b4b2-d5fe939b85cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:34:11.185472 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.185450    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a46c4527-754c-4c7d-b4b2-d5fe939b85cd" (UID: "a46c4527-754c-4c7d-b4b2-d5fe939b85cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:34:11.185623 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.185469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj" (OuterVolumeSpecName: "kube-api-access-kc5tj") pod "a46c4527-754c-4c7d-b4b2-d5fe939b85cd" (UID: "a46c4527-754c-4c7d-b4b2-d5fe939b85cd"). InnerVolumeSpecName "kube-api-access-kc5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:34:11.190896 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.190864    2580 scope.go:117] "RemoveContainer" containerID="efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"
Sep 11 21:34:11.191178 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:34:11.191158    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7\": container with ID starting with efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7 not found: ID does not exist" containerID="efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"
Sep 11 21:34:11.191245 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.191189    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7"} err="failed to get container status \"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7\": rpc error: code = NotFound desc = could not find container \"efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7\": container with ID starting with efc0d03723d8839b9350053f88434fbf075810f48daa67b4dd1837704e32c3c7 not found: ID does not exist"
Sep 11 21:34:11.284683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.284637    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:11.284683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.284671    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:11.284683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.284680    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:11.284683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.284689    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5tj\" (UniqueName: \"kubernetes.io/projected/a46c4527-754c-4c7d-b4b2-d5fe939b85cd-kube-api-access-kc5tj\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:11.504165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.504135    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:34:11.510381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:11.510361    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c0162b-6dcff977f-mr4pc"]
Sep 11 21:34:12.385197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:12.385163    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" path="/var/lib/kubelet/pods/a46c4527-754c-4c7d-b4b2-d5fe939b85cd/volumes"
Sep 11 21:34:15.145672 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.145626    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" podUID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" containerName="main" containerID="cri-o://82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900" gracePeriod=45
Sep 11 21:34:15.408293 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.408242    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:34:15.522776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.522743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.522963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.522798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.522963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.522854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.522963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.522893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.522963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.522925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.523171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523000    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txt87\" (UniqueName: \"kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87\") pod \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\" (UID: \"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7\") "
Sep 11 21:34:15.523171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:15.523287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home" (OuterVolumeSpecName: "home") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:15.523287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523198    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:15.523372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523331    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.523372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523351    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.523372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.523361    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.525258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.525234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm" (OuterVolumeSpecName: "dshm") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:15.525258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.525237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:34:15.525404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.525256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87" (OuterVolumeSpecName: "kube-api-access-txt87") pod "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" (UID: "13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7"). InnerVolumeSpecName "kube-api-access-txt87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:34:15.624533 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.624489    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txt87\" (UniqueName: \"kubernetes.io/projected/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-kube-api-access-txt87\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.624639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.624537    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.624639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.624552    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:34:15.989916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:15.989884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:34:16.209278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.209243    2580 generic.go:356] "Generic (PLEG): container finished" podID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" containerID="82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900" exitCode=0
Sep 11 21:34:16.209864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.209302    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"
Sep 11 21:34:16.209864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.209348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" event={"ID":"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7","Type":"ContainerDied","Data":"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"}
Sep 11 21:34:16.209864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.209409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s" event={"ID":"13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7","Type":"ContainerDied","Data":"b414a49d2e3db9d1a04abcd87791bc6e690648db4a2a9507793e38ae440b5099"}
Sep 11 21:34:16.209864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.209437    2580 scope.go:117] "RemoveContainer" containerID="82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"
Sep 11 21:34:16.219383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.219366    2580 scope.go:117] "RemoveContainer" containerID="82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"
Sep 11 21:34:16.219730 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:34:16.219708    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900\": container with ID starting with 82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900 not found: ID does not exist" containerID="82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"
Sep 11 21:34:16.219852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.219738    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900"} err="failed to get container status \"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900\": rpc error: code = NotFound desc = could not find container \"82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900\": container with ID starting with 82d64429b7fdbf9d6a2a47c57aaf9b487d2eadd6fc5dc69fdd82adc77f323900 not found: ID does not exist"
Sep 11 21:34:16.233515 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.233475    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:34:16.237141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.237120    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c0162b/explicit-model-c0162b-kserve-6957b894cc-6fc8s"]
Sep 11 21:34:16.384733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:16.384648    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" path="/var/lib/kubelet/pods/13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7/volumes"
Sep 11 21:34:19.476806 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.476770    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:34:19.477311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477291    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" containerName="main"
Sep 11 21:34:19.477393 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477315    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" containerName="main"
Sep 11 21:34:19.477393 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477361    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" containerName="maas-api"
Sep 11 21:34:19.477393 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477370    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" containerName="maas-api"
Sep 11 21:34:19.477570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477455    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13110bb9-bfc2-4b6b-ad76-0ced2bdb69f7" containerName="main"
Sep 11 21:34:19.477570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.477471    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46c4527-754c-4c7d-b4b2-d5fe939b85cd" containerName="maas-api"
Sep 11 21:34:19.482339 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.482317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.484754 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.484734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-85f1ab-openshift-default-dockercfg-7blb6\""
Sep 11 21:34:19.498087 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.497979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:34:19.562411 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bhd\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.562766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.562717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663595 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663595 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.663849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.663991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bhd\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.664011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.664048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664314 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.664072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664380 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.664363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.664562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.664541    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.666197 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.666175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.666546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.666529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.671692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.671668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bhd\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.672238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.672217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token\") pod \"e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") " pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:19.795308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:19.795288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:20.076285 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.076259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:34:20.078371 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:34:20.078346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd355f_37dd_4012_b501_a4e53a6585e0.slice/crio-4c47c8c42885309a8ea4b87d2571fe341b96472f864e953e2bb8920a7f15a369 WatchSource:0}: Error finding container 4c47c8c42885309a8ea4b87d2571fe341b96472f864e953e2bb8920a7f15a369: Status 404 returned error can't find the container with id 4c47c8c42885309a8ea4b87d2571fe341b96472f864e953e2bb8920a7f15a369
Sep 11 21:34:20.080956 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.080925    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:34:20.081012 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.080991    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:34:20.081051 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.081019    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:34:20.234777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.234742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" event={"ID":"9bcd355f-37dd-4012-b501-a4e53a6585e0","Type":"ContainerStarted","Data":"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"}
Sep 11 21:34:20.234946 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.234806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" event={"ID":"9bcd355f-37dd-4012-b501-a4e53a6585e0","Type":"ContainerStarted","Data":"4c47c8c42885309a8ea4b87d2571fe341b96472f864e953e2bb8920a7f15a369"}
Sep 11 21:34:20.256086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.256033    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" podStartSLOduration=1.256016668 podStartE2EDuration="1.256016668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:34:20.25390208 +0000 UTC m=+1008.376234001" watchObservedRunningTime="2026-09-11 21:34:20.256016668 +0000 UTC m=+1008.378348587"
Sep 11 21:34:20.795534 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:20.795488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:21.800489 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:21.800461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:22.246726 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:22.246700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:22.247678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:22.247660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:34:40.396910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:40.396876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:34:57.773965 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.773935    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:34:57.778879 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.778860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.781381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.781363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"distinct-w0-db06d6-kserve-self-signed-certs\""
Sep 11 21:34:57.788891 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.788868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:34:57.886304 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljrc\" (UniqueName: \"kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.886426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.886426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.886426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.886576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.886576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.886493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988045 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljrc\" (UniqueName: \"kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988423 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988601 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988723 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.988789 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.988770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.990602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.990573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.991020 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.990998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:57.996000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:57.995979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljrc\" (UniqueName: \"kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc\") pod \"distinct-w0-db06d6-kserve-86b45fdb8f-npvcw\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:58.088805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:58.088745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:34:58.364812 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:58.364777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:34:58.367112 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:34:58.367087    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2992f556_45ab_4435_8b97_940dc68e803e.slice/crio-24b3ea5fcdf1955632b134f2d2769c0aea0b4d4fd2b70d92890e93ccabde76fe WatchSource:0}: Error finding container 24b3ea5fcdf1955632b134f2d2769c0aea0b4d4fd2b70d92890e93ccabde76fe: Status 404 returned error can't find the container with id 24b3ea5fcdf1955632b134f2d2769c0aea0b4d4fd2b70d92890e93ccabde76fe
Sep 11 21:34:58.437388 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:58.437354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" event={"ID":"2992f556-45ab-4435-8b97-940dc68e803e","Type":"ContainerStarted","Data":"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"}
Sep 11 21:34:58.437524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:58.437406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" event={"ID":"2992f556-45ab-4435-8b97-940dc68e803e","Type":"ContainerStarted","Data":"24b3ea5fcdf1955632b134f2d2769c0aea0b4d4fd2b70d92890e93ccabde76fe"}
Sep 11 21:34:58.481233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:34:58.481180    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" podStartSLOduration=1.4811628620000001 podStartE2EDuration="1.481162862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:34:58.477983782 +0000 UTC m=+1046.600315701" watchObservedRunningTime="2026-09-11 21:34:58.481162862 +0000 UTC m=+1046.603494781"
Sep 11 21:35:07.793426 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:07.793386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:35:08.089942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:08.089845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:35:08.090209 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:08.090187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:35:08.116037 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:08.116006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:35:08.505065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:08.505037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:35:23.273747 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:23.273716    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:35:23.274154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:23.273935    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" podUID="9bcd355f-37dd-4012-b501-a4e53a6585e0" containerName="istio-proxy" containerID="cri-o://82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6" gracePeriod=30
Sep 11 21:35:25.170440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:25.170407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:35:28.534067 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.534008    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:35:28.573610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7bhd\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573735    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573795    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.573961 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token\") pod \"9bcd355f-37dd-4012-b501-a4e53a6585e0\" (UID: \"9bcd355f-37dd-4012-b501-a4e53a6585e0\") "
Sep 11 21:35:28.574164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.573983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:28.574164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:28.574307 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574285    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd355f-37dd-4012-b501-a4e53a6585e0-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.574373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574314    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.574373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:28.574373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:28.574547 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.574474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:28.576407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.576385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:35:28.576666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.576641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:28.576745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.576709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:28.576745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.576725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd" (OuterVolumeSpecName: "kube-api-access-j7bhd") pod "9bcd355f-37dd-4012-b501-a4e53a6585e0" (UID: "9bcd355f-37dd-4012-b501-a4e53a6585e0"). InnerVolumeSpecName "kube-api-access-j7bhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:28.605167 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.605138    2580 generic.go:356] "Generic (PLEG): container finished" podID="9bcd355f-37dd-4012-b501-a4e53a6585e0" containerID="82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6" exitCode=0
Sep 11 21:35:28.605257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.605186    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"
Sep 11 21:35:28.605257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.605225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" event={"ID":"9bcd355f-37dd-4012-b501-a4e53a6585e0","Type":"ContainerDied","Data":"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"}
Sep 11 21:35:28.605365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.605273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c" event={"ID":"9bcd355f-37dd-4012-b501-a4e53a6585e0","Type":"ContainerDied","Data":"4c47c8c42885309a8ea4b87d2571fe341b96472f864e953e2bb8920a7f15a369"}
Sep 11 21:35:28.605365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.605296    2580 scope.go:117] "RemoveContainer" containerID="82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"
Sep 11 21:35:28.613995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.613980    2580 scope.go:117] "RemoveContainer" containerID="82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"
Sep 11 21:35:28.614286 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:35:28.614268    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6\": container with ID starting with 82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6 not found: ID does not exist" containerID="82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"
Sep 11 21:35:28.614337 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.614294    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6"} err="failed to get container status \"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6\": rpc error: code = NotFound desc = could not find container \"82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6\": container with ID starting with 82b807b3f3241e71705d2a8dc42e29537a472c74ce8df0b9e77ded59205be3d6 not found: ID does not exist"
Sep 11 21:35:28.631654 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.631624    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:35:28.635725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.635699    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-85f1ab-openshift-default-5c7864cbb-rqj7c"]
Sep 11 21:35:28.674925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674899    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674932    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674946    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7bhd\" (UniqueName: \"kubernetes.io/projected/9bcd355f-37dd-4012-b501-a4e53a6585e0-kube-api-access-j7bhd\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674959    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674970    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674980    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:28.675046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:28.674992    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd355f-37dd-4012-b501-a4e53a6585e0-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:30.383938 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:30.383910    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcd355f-37dd-4012-b501-a4e53a6585e0" path="/var/lib/kubelet/pods/9bcd355f-37dd-4012-b501-a4e53a6585e0/volumes"
Sep 11 21:35:32.857729 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.857694    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:35:32.858283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.858261    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcd355f-37dd-4012-b501-a4e53a6585e0" containerName="istio-proxy"
Sep 11 21:35:32.858451 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.858428    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcd355f-37dd-4012-b501-a4e53a6585e0" containerName="istio-proxy"
Sep 11 21:35:32.858593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.858571    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bcd355f-37dd-4012-b501-a4e53a6585e0" containerName="istio-proxy"
Sep 11 21:35:32.863233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.863212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.865688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.865664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-408219-openshift-default-dockercfg-fxwdr\""
Sep 11 21:35:32.872933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.872911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:35:32.907068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907237 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907237 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907467 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhbf\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:32.907557 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:32.907467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhbf\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.009744 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.009405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.010115 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.010088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.010115 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.010111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.010528 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.010462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.010779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.010736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.010844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.010813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.012228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.012197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.012380 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.012360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.018731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.018704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.019047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.019026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhbf\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf\") pod \"e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") " pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.175123 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.175065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:33.452802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.452779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:35:33.454886 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:35:33.454862    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda830f849_1a60_440f_a19e_9b4a4a8db382.slice/crio-0622ca5da874e86a4952d2a389bc25308edef6e1608a263c46469f6a6d154566 WatchSource:0}: Error finding container 0622ca5da874e86a4952d2a389bc25308edef6e1608a263c46469f6a6d154566: Status 404 returned error can't find the container with id 0622ca5da874e86a4952d2a389bc25308edef6e1608a263c46469f6a6d154566
Sep 11 21:35:33.457234 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.457203    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:35:33.457308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.457265    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:35:33.457308 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.457291    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:35:33.635599 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.635560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerStarted","Data":"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"}
Sep 11 21:35:33.635783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.635627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerStarted","Data":"0622ca5da874e86a4952d2a389bc25308edef6e1608a263c46469f6a6d154566"}
Sep 11 21:35:33.654730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:33.654677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podStartSLOduration=1.654662916 podStartE2EDuration="1.654662916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:35:33.653552488 +0000 UTC m=+1081.775884408" watchObservedRunningTime="2026-09-11 21:35:33.654662916 +0000 UTC m=+1081.776994835"
Sep 11 21:35:34.175968 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:34.175931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:34.181014 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:34.180991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:34.640981 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:34.640949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:34.641950 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:34.641931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:35:35.461683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:35.461654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:35:40.189634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.189578    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" podUID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" containerName="main" containerID="cri-o://95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091" gracePeriod=45
Sep 11 21:35:40.428381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.428354    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:35:40.479135 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479158    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5hxj\" (UniqueName: \"kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479217    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache\") pod \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\" (UID: \"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9\") "
Sep 11 21:35:40.479450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home" (OuterVolumeSpecName: "home") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:40.479450 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:40.479716 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:40.479990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479814    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.479990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479838    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.479990 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.479852    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.481668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.481644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj" (OuterVolumeSpecName: "kube-api-access-s5hxj") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "kube-api-access-s5hxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:40.482048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.482011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm" (OuterVolumeSpecName: "dshm") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:40.482048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.482011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" (UID: "e444a08a-34b2-4cdb-9d6e-f0460e85e7a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:35:40.580560 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.580525    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5hxj\" (UniqueName: \"kubernetes.io/projected/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-kube-api-access-s5hxj\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.580560 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.580555    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.580560 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.580564    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:40.673927 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.673896    2580 generic.go:356] "Generic (PLEG): container finished" podID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" containerID="95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091" exitCode=0
Sep 11 21:35:40.674063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.673942    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"
Sep 11 21:35:40.674063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.673954    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" event={"ID":"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9","Type":"ContainerDied","Data":"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"}
Sep 11 21:35:40.674063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.673994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk" event={"ID":"e444a08a-34b2-4cdb-9d6e-f0460e85e7a9","Type":"ContainerDied","Data":"579799d93746b596780de677b8ddacd3e2adfe170d33a0cb62d6db51a9fe5b08"}
Sep 11 21:35:40.674063 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.674017    2580 scope.go:117] "RemoveContainer" containerID="95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"
Sep 11 21:35:40.683263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.683247    2580 scope.go:117] "RemoveContainer" containerID="95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"
Sep 11 21:35:40.683571 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:35:40.683552    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091\": container with ID starting with 95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091 not found: ID does not exist" containerID="95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"
Sep 11 21:35:40.683655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.683582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091"} err="failed to get container status \"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091\": rpc error: code = NotFound desc = could not find container \"95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091\": container with ID starting with 95ea310f1e5c7ef6eac5b559574400a118b4e0bf38515f2933d7b876ca4e0091 not found: ID does not exist"
Sep 11 21:35:40.696983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.696961    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:35:40.700543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:40.700521    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1a1e3a/test-model-1a1e3a-kserve-754fbbdd4f-n66tk"]
Sep 11 21:35:42.383196 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:42.383163    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" path="/var/lib/kubelet/pods/e444a08a-34b2-4cdb-9d6e-f0460e85e7a9/volumes"
Sep 11 21:35:43.190721 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:43.190687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:35:50.481332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.481280    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" podUID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" containerName="main" containerID="cri-o://c314128c2361f2550cfc25b0c5616a02f24d2f07ba84352475bcb59a32c37e4f" gracePeriod=45
Sep 11 21:35:50.733449 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.733374    2580 generic.go:356] "Generic (PLEG): container finished" podID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" containerID="c314128c2361f2550cfc25b0c5616a02f24d2f07ba84352475bcb59a32c37e4f" exitCode=0
Sep 11 21:35:50.733571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.733449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" event={"ID":"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e","Type":"ContainerDied","Data":"c314128c2361f2550cfc25b0c5616a02f24d2f07ba84352475bcb59a32c37e4f"}
Sep 11 21:35:50.733571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.733491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq" event={"ID":"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e","Type":"ContainerDied","Data":"1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7"}
Sep 11 21:35:50.733571 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.733529    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ffd4b93c2e726f4d7d0cf995f9a2ba641f06170f7242248bf6fae652d0368b7"
Sep 11 21:35:50.740087 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.740067    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:35:50.874733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-268r9\" (UniqueName: \"kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir\") pod \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\" (UID: \"95051592-ffd6-4a5e-bf2f-a9e3ae28a59e\") "
Sep 11 21:35:50.874995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.874980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache" (OuterVolumeSpecName: "model-cache") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:50.875251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.875010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home" (OuterVolumeSpecName: "home") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:50.875251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.875156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:50.875251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.875226    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:50.875251 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.875240    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:50.877136 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.877111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm" (OuterVolumeSpecName: "dshm") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:50.877242 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.877142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9" (OuterVolumeSpecName: "kube-api-access-268r9") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "kube-api-access-268r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:50.877242 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.877177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" (UID: "95051592-ffd6-4a5e-bf2f-a9e3ae28a59e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:35:50.976249 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.976220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268r9\" (UniqueName: \"kubernetes.io/projected/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-kube-api-access-268r9\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:50.976249 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.976242    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:50.976249 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.976251    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:50.976421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:50.976259    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:51.739309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:51.739279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"
Sep 11 21:35:51.760465 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:51.760441    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:35:51.764370 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:51.764348    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-02c60f/test-model-02c60f-kserve-6b7d844587-sgqvq"]
Sep 11 21:35:52.389717 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:52.389686    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" path="/var/lib/kubelet/pods/95051592-ffd6-4a5e-bf2f-a9e3ae28a59e/volumes"
Sep 11 21:35:53.763304 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:53.763268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:35:53.763785 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:53.763571    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy" containerID="cri-o://d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f" gracePeriod=30
Sep 11 21:35:54.176389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"]
Sep 11 21:35:54.176678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" containerName="main"
Sep 11 21:35:54.176737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176679    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" containerName="main"
Sep 11 21:35:54.176737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176701    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" containerName="main"
Sep 11 21:35:54.176737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176707    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" containerName="main"
Sep 11 21:35:54.176831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176766    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="95051592-ffd6-4a5e-bf2f-a9e3ae28a59e" containerName="main"
Sep 11 21:35:54.176831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.176777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e444a08a-34b2-4cdb-9d6e-f0460e85e7a9" containerName="main"
Sep 11 21:35:54.179922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.179907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:54.184907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.184882    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nhbxk\""
Sep 11 21:35:54.185053 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.184882    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:35:54.219908 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.219880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"]
Sep 11 21:35:54.307536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.307491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs\" (UID: \"e32d0236-5ed8-460a-9007-3d7e50eb3c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:54.408784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.408754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs\" (UID: \"e32d0236-5ed8-460a-9007-3d7e50eb3c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:54.409342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.409319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs\" (UID: \"e32d0236-5ed8-460a-9007-3d7e50eb3c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:54.491153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.491128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:54.768610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.768539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"]
Sep 11 21:35:54.771611 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:35:54.771586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32d0236_5ed8_460a_9007_3d7e50eb3c57.slice/crio-e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47 WatchSource:0}: Error finding container e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47: Status 404 returned error can't find the container with id e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47
Sep 11 21:35:54.989206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.989169    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:35:54.993665 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.993646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:54.996311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:54.996293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"e2e-conflict-b-315f74cf-kserve-self-signed-certs\""
Sep 11 21:35:55.004042 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.004023    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:35:55.117234 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7649\" (UniqueName: \"kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.117403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.117403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.117403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.117403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.117566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.117464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218059 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7649\" (UniqueName: \"kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218606 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.218889 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.218722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.220385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.220367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.220767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.220751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.226468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.226442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7649\" (UniqueName: \"kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649\") pod \"e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") " pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.303060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.302955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:35:55.574620 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.574599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:35:55.576256 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:35:55.576236    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce492243_3c20_486c_92a5_586f1dae9c4d.slice/crio-1323759f9b74ec25444c78a65e06d35a71eac0c41dafa78cee16dcc8c3c33da3 WatchSource:0}: Error finding container 1323759f9b74ec25444c78a65e06d35a71eac0c41dafa78cee16dcc8c3c33da3: Status 404 returned error can't find the container with id 1323759f9b74ec25444c78a65e06d35a71eac0c41dafa78cee16dcc8c3c33da3
Sep 11 21:35:55.768624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.768586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" event={"ID":"ce492243-3c20-486c-92a5-586f1dae9c4d","Type":"ContainerStarted","Data":"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"}
Sep 11 21:35:55.768624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.768632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" event={"ID":"ce492243-3c20-486c-92a5-586f1dae9c4d","Type":"ContainerStarted","Data":"1323759f9b74ec25444c78a65e06d35a71eac0c41dafa78cee16dcc8c3c33da3"}
Sep 11 21:35:55.770114 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.770091    2580 generic.go:356] "Generic (PLEG): container finished" podID="e32d0236-5ed8-460a-9007-3d7e50eb3c57" containerID="94b01f19c9440ae119fe730f36e8ee70b30111219f4b40684c0cdcdfbefa7527" exitCode=0
Sep 11 21:35:55.770219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.770128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs" event={"ID":"e32d0236-5ed8-460a-9007-3d7e50eb3c57","Type":"ContainerDied","Data":"94b01f19c9440ae119fe730f36e8ee70b30111219f4b40684c0cdcdfbefa7527"}
Sep 11 21:35:55.770219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.770200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs" event={"ID":"e32d0236-5ed8-460a-9007-3d7e50eb3c57","Type":"ContainerStarted","Data":"e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47"}
Sep 11 21:35:55.783974 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:55.783922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" podStartSLOduration=1.783909889 podStartE2EDuration="1.783909889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:35:55.783743455 +0000 UTC m=+1103.906075368" watchObservedRunningTime="2026-09-11 21:35:55.783909889 +0000 UTC m=+1103.906241841"
Sep 11 21:35:56.899940 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:56.899917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:57.037336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.037257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca\") pod \"e32d0236-5ed8-460a-9007-3d7e50eb3c57\" (UID: \"e32d0236-5ed8-460a-9007-3d7e50eb3c57\") "
Sep 11 21:35:57.037573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.037551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e32d0236-5ed8-460a-9007-3d7e50eb3c57" (UID: "e32d0236-5ed8-460a-9007-3d7e50eb3c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:57.138108 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.138081    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32d0236-5ed8-460a-9007-3d7e50eb3c57-maas-api-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:57.784580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.784539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs" event={"ID":"e32d0236-5ed8-460a-9007-3d7e50eb3c57","Type":"ContainerDied","Data":"e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47"}
Sep 11 21:35:57.784758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.784607    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a57d3e865daca2d52ae8dddbe8659c9f0876d7977f642383a061a01752ab47"
Sep 11 21:35:57.784758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:57.784659    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"
Sep 11 21:35:59.015761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.015739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:35:59.158046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.157975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158046 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158016    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pwfv\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158233 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158311    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs\") pod \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\" (UID: \"c8eba0fd-0581-41cd-bd97-2c76791bbf25\") "
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158736    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158752    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158765    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158778    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.158818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.158790    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.160523 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.160487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:59.160638 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.160595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv" (OuterVolumeSpecName: "kube-api-access-4pwfv") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "kube-api-access-4pwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:59.160886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.160870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:59.160886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.160869    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8eba0fd-0581-41cd-bd97-2c76791bbf25" (UID: "c8eba0fd-0581-41cd-bd97-2c76791bbf25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:35:59.259299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.259268    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.259299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.259293    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.259299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.259301    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8eba0fd-0581-41cd-bd97-2c76791bbf25-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.259552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.259309    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwfv\" (UniqueName: \"kubernetes.io/projected/c8eba0fd-0581-41cd-bd97-2c76791bbf25-kube-api-access-4pwfv\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:35:59.799227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.799194    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerID="d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f" exitCode=0
Sep 11 21:35:59.799374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.799241    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"
Sep 11 21:35:59.799374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.799272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerDied","Data":"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"}
Sep 11 21:35:59.799374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.799322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7" event={"ID":"c8eba0fd-0581-41cd-bd97-2c76791bbf25","Type":"ContainerDied","Data":"da8a0f6cadc6e6facc9380e0f62b4af23bb3ea01befa66df2df6d56d58631c47"}
Sep 11 21:35:59.799374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.799345    2580 scope.go:117] "RemoveContainer" containerID="d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"
Sep 11 21:35:59.809406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.809390    2580 scope.go:117] "RemoveContainer" containerID="26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"
Sep 11 21:35:59.817289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.817271    2580 scope.go:117] "RemoveContainer" containerID="d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"
Sep 11 21:35:59.817601 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:35:59.817576    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f\": container with ID starting with d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f not found: ID does not exist" containerID="d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"
Sep 11 21:35:59.817714 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.817609    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f"} err="failed to get container status \"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f\": rpc error: code = NotFound desc = could not find container \"d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f\": container with ID starting with d05b4bbbb36e2f1412de2adcb2109939f688fad80f582d0e6e7c4bed8f9c165f not found: ID does not exist"
Sep 11 21:35:59.817714 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.817632    2580 scope.go:117] "RemoveContainer" containerID="26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"
Sep 11 21:35:59.817909 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:35:59.817892    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1\": container with ID starting with 26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1 not found: ID does not exist" containerID="26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"
Sep 11 21:35:59.817962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.817915    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1"} err="failed to get container status \"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1\": rpc error: code = NotFound desc = could not find container \"26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1\": container with ID starting with 26c624582cb3ed11b0b903bbdd802b09fb54cb81346c5319979b3d39ef8799e1 not found: ID does not exist"
Sep 11 21:35:59.821035 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.821014    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:35:59.828414 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:35:59.828385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1a1e3a-openshift-default-64969f57f8-wtcx7"]
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.245840    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246580    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246592    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32d0236-5ed8-460a-9007-3d7e50eb3c57" containerName="revoke-keys"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32d0236-5ed8-460a-9007-3d7e50eb3c57" containerName="revoke-keys"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246636    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246646    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246731    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32d0236-5ed8-460a-9007-3d7e50eb3c57" containerName="revoke-keys"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246743    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.250525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.246751    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" containerName="istio-proxy"
Sep 11 21:36:00.251677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.251653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.261277 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.261252    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc0b206e-openshift-default-dockercfg-vnfg7\""
Sep 11 21:36:00.271855 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.271775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:36:00.370826 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.370782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.370888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.370936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.370984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.371029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.371104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.371140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.371169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.371440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.371298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497nm\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.387996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.387960    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8eba0fd-0581-41cd-bd97-2c76791bbf25" path="/var/lib/kubelet/pods/c8eba0fd-0581-41cd-bd97-2c76791bbf25/volumes"
Sep 11 21:36:00.472610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.472844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497nm\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.472986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.473019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.473089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.473116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.473782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.473694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.475318 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.475295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.475716 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.475698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.482628 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.482608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.483487 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.483470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-497nm\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm\") pod \"e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") " pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.578549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.578446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:00.873605 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.873578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:36:00.875339 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:36:00.875311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd424bf60_80ee_4c9a_85c7_680efd69a0be.slice/crio-ee1fe2fbfdfefd8d73d845d779ef3edca49121f5b0db1c6b95da400ed1fa66c1 WatchSource:0}: Error finding container ee1fe2fbfdfefd8d73d845d779ef3edca49121f5b0db1c6b95da400ed1fa66c1: Status 404 returned error can't find the container with id ee1fe2fbfdfefd8d73d845d779ef3edca49121f5b0db1c6b95da400ed1fa66c1
Sep 11 21:36:00.877555 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.877493    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:00.877658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.877587    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:00.877658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:00.877621    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:01.812292 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:01.812255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerStarted","Data":"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"}
Sep 11 21:36:01.812741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:01.812315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerStarted","Data":"ee1fe2fbfdfefd8d73d845d779ef3edca49121f5b0db1c6b95da400ed1fa66c1"}
Sep 11 21:36:01.838905 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:01.838843    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podStartSLOduration=1.838823096 podStartE2EDuration="1.838823096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:01.836003621 +0000 UTC m=+1109.958335554" watchObservedRunningTime="2026-09-11 21:36:01.838823096 +0000 UTC m=+1109.961155017"
Sep 11 21:36:02.578777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:02.578747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:02.584163 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:02.584137    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:02.818080 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:02.818049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:02.818964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:02.818946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:05.303385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:05.303347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:36:05.303769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:05.303408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:36:05.315767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:05.315743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:36:05.847468 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:05.847439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:36:07.642539 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:07.642483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"]
Sep 11 21:36:07.648656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:07.648630    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02c60f-9stbs"]
Sep 11 21:36:08.384525 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.384466    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32d0236-5ed8-460a-9007-3d7e50eb3c57" path="/var/lib/kubelet/pods/e32d0236-5ed8-460a-9007-3d7e50eb3c57/volumes"
Sep 11 21:36:08.855351 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.855318    2580 generic.go:356] "Generic (PLEG): container finished" podID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerID="4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549" exitCode=0
Sep 11 21:36:08.855757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.855395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerDied","Data":"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"}
Sep 11 21:36:08.855831 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.855810    2580 scope.go:117] "RemoveContainer" containerID="4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"
Sep 11 21:36:08.856878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.856844    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:08.856991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.856930    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:08.856991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:08.856978    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:09.864807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:09.864766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerStarted","Data":"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"}
Sep 11 21:36:10.019447 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:10.019411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:36:10.175358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:10.175273    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:36:10.175619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:10.175591    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:36:10.175693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:10.175656    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 21:36:11.176427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:11.176391    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:36:11.176850 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:11.176462    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 21:36:12.176416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:12.176381    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:36:12.176625 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:12.176434    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 21:36:13.176328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:13.176289    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:36:13.176554 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:13.176352    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 21:36:14.176280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:14.176247    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:36:14.176698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:14.176305    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 21:36:14.207096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:14.207065    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:36:14.207363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:14.207340    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" containerID="cri-o://53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff" gracePeriod=30
Sep 11 21:36:15.180255 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:15.180226    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:36:15.180642 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:15.180491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:36:15.181243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:15.181223    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:36:18.685651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:18.685617    2580 patch_prober.go:28] interesting pod/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:36:18.686000 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:18.685674    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:36:19.456900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.456871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:36:19.573135 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:19.573482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bfxt\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt\") pod \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\" (UID: \"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c\") "
Sep 11 21:36:19.573774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573523    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:19.573774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:19.573774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:19.573940 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573807    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.573940 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573826    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.573940 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573839    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.573940 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.573901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:36:19.575483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.575460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:36:19.575632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.575615    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:36:19.575782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.575762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:36:19.575866 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.575852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt" (OuterVolumeSpecName: "kube-api-access-7bfxt") pod "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" (UID: "0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c"). InnerVolumeSpecName "kube-api-access-7bfxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675169    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675201    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675213    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675223    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675234    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.675270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.675246    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bfxt\" (UniqueName: \"kubernetes.io/projected/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c-kube-api-access-7bfxt\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:36:19.920535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.920419    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerID="53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff" exitCode=0
Sep 11 21:36:19.920535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.920478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"
Sep 11 21:36:19.921031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.920538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerDied","Data":"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"}
Sep 11 21:36:19.921031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.920594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f" event={"ID":"0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c","Type":"ContainerDied","Data":"08d5cbf09b299b169fe1baac2da4eb208c9471d2fde9b580617c6654683d930c"}
Sep 11 21:36:19.921031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.920617    2580 scope.go:117] "RemoveContainer" containerID="53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"
Sep 11 21:36:19.930782 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.930751    2580 scope.go:117] "RemoveContainer" containerID="3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"
Sep 11 21:36:19.938915 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.938892    2580 scope.go:117] "RemoveContainer" containerID="53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"
Sep 11 21:36:19.939336 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:19.939263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff\": container with ID starting with 53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff not found: ID does not exist" containerID="53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"
Sep 11 21:36:19.939336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.939303    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff"} err="failed to get container status \"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff\": rpc error: code = NotFound desc = could not find container \"53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff\": container with ID starting with 53b6a6106f2fa0573369546bda01076e4a4500d41f97e0a2f64fff56c4ead8ff not found: ID does not exist"
Sep 11 21:36:19.939336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.939327    2580 scope.go:117] "RemoveContainer" containerID="3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"
Sep 11 21:36:19.939790 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:19.939764    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7\": container with ID starting with 3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7 not found: ID does not exist" containerID="3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"
Sep 11 21:36:19.939912 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.939799    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7"} err="failed to get container status \"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7\": rpc error: code = NotFound desc = could not find container \"3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7\": container with ID starting with 3b2f5e52d4cd8eb2abb7582cea06f9ffa4a066d6c6d1cff703eb259485f39eb7 not found: ID does not exist"
Sep 11 21:36:19.941639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.941620    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:36:19.945946 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:19.945927    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-02c60f-openshift-default-848bdbd574-kd74f"]
Sep 11 21:36:20.384521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:20.384462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" path="/var/lib/kubelet/pods/0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c/volumes"
Sep 11 21:36:23.908263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908226    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:36:23.908796 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908774    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.908796 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908795    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.908983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.908983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908828    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.908983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908931    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.908983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.908941    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecb8aa0-11fa-49bd-819f-b6a9d2f1396c" containerName="istio-proxy"
Sep 11 21:36:23.913985 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.913960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:23.916784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.916763    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:36:23.917029 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.917012    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czlrt\""
Sep 11 21:36:23.918132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.918029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:36:23.918132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.918115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c97420-serving-cert\""
Sep 11 21:36:23.918313 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.918146    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:36:23.923486 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:23.923238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:36:24.015066 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.015026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.015208 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.015102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94v4q\" (UniqueName: \"kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.015208 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.015139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.015297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.015219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.116274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.116235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94v4q\" (UniqueName: \"kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.116441 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.116281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.116441 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.116324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.116441 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.116380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.116724 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:24.116497    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-serving-cert: secret "maas-api-e2e-shared-a-w3-c97420-serving-cert" not found
Sep 11 21:36:24.116724 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:24.116600    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls podName:49f20de6-f34d-4d72-a355-24a344334f6e nodeName:}" failed. No retries permitted until 2026-09-11 21:36:24.616576874 +0000 UTC m=+1132.738908776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e") : secret "maas-api-e2e-shared-a-w3-c97420-serving-cert" not found
Sep 11 21:36:24.117037 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.117011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.118903 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.118874    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.126359 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.126339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94v4q\" (UniqueName: \"kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.622383 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:24.622353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:24.622567 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:24.622489    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-serving-cert: secret "maas-api-e2e-shared-a-w3-c97420-serving-cert" not found
Sep 11 21:36:24.622567 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:24.622565    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls podName:49f20de6-f34d-4d72-a355-24a344334f6e nodeName:}" failed. No retries permitted until 2026-09-11 21:36:25.622551073 +0000 UTC m=+1133.744882971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e") : secret "maas-api-e2e-shared-a-w3-c97420-serving-cert" not found
Sep 11 21:36:25.633702 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:25.633665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:25.636119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:25.636095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:25.741091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:25.741060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:26.014299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:26.014269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:36:26.968785 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:26.968757    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" event={"ID":"49f20de6-f34d-4d72-a355-24a344334f6e","Type":"ContainerStarted","Data":"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"}
Sep 11 21:36:26.969146 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:26.968803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" event={"ID":"49f20de6-f34d-4d72-a355-24a344334f6e","Type":"ContainerStarted","Data":"9cf38e6f6779f0cf9eaa15e853cebc623b00b9b21f8632cd677ca2d6c0dfd1f4"}
Sep 11 21:36:26.969146 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:26.968900    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:26.987880 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:26.987832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" podStartSLOduration=3.666321002 podStartE2EDuration="3.987816326s" totalImagesPullingTime="321.495324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:26.986347858 +0000 UTC m=+1135.108679776" watchObservedRunningTime="2026-09-11 21:36:26.987816326 +0000 UTC m=+1135.110148244"
Sep 11 21:36:32.450275 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:32.450246    2580 scope.go:117] "RemoveContainer" containerID="fd21884ef99ba1a9bc4018d5d5323c964c21d932dda076ed8c6ac76753a186c3"
Sep 11 21:36:32.459008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:32.458985    2580 scope.go:117] "RemoveContainer" containerID="2b7a46ceaa57be68f1277c7ec46e35b9a5a548271ea99b01d26952d79e4a4ebd"
Sep 11 21:36:32.980409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:32.980382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:36:33.860660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:33.860624    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 21:36:33.861014 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:33.860696    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 21:36:35.013011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.012981    2580 generic.go:356] "Generic (PLEG): container finished" podID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerID="012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4" exitCode=0
Sep 11 21:36:35.013418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.013040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerDied","Data":"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"}
Sep 11 21:36:35.013418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.013372    2580 scope.go:117] "RemoveContainer" containerID="012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"
Sep 11 21:36:35.014473 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.014408    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:35.014612 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.014494    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:35.014612 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:35.014570    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:36:36.020296 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.020264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerStarted","Data":"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"}
Sep 11 21:36:36.335692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.335605    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:36:36.348281 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.348252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.358524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.356036    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-db06d6\"/\"distinct-2-w0-db06d6-kserve-self-signed-certs\""
Sep 11 21:36:36.363423 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.363397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:36:36.452997 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.452965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.453176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.453122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.453252 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.453210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.453307 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.453257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.453371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.453332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.453536 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.453487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjjb\" (UniqueName: \"kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.554341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.554305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.555037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.555227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.555368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.554936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555699 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.555599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjjb\" (UniqueName: \"kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.555923 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.555867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.556146 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.556127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.556379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.556355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.558524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.558027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.558607 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.558548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.564437 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.564415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjjb\" (UniqueName: \"kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb\") pod \"distinct-2-w0-db06d6-kserve-bf786876b-k6qjh\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") " pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.579214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.579157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:36.579401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.579379    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 21:36:36.579443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.579428    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 21:36:36.671698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.671630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:36.966532 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:36.965992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:36:36.967727 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:36:36.967693    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf227fd7c_24a0_4e71_93b7_4e4bc0f3b646.slice/crio-9837a099eefde3719ae0b10f2d47faeccb459dfcc702bf08f19aa6a39b90d92b WatchSource:0}: Error finding container 9837a099eefde3719ae0b10f2d47faeccb459dfcc702bf08f19aa6a39b90d92b: Status 404 returned error can't find the container with id 9837a099eefde3719ae0b10f2d47faeccb459dfcc702bf08f19aa6a39b90d92b
Sep 11 21:36:37.027305 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:37.027279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" event={"ID":"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646","Type":"ContainerStarted","Data":"9837a099eefde3719ae0b10f2d47faeccb459dfcc702bf08f19aa6a39b90d92b"}
Sep 11 21:36:37.579172 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:37.579137    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 21:36:37.579330 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:37.579190    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 21:36:38.038270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:38.038230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" event={"ID":"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646","Type":"ContainerStarted","Data":"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"}
Sep 11 21:36:38.060389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:38.060326    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" podStartSLOduration=2.06030693 podStartE2EDuration="2.06030693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:38.056153885 +0000 UTC m=+1146.178485797" watchObservedRunningTime="2026-09-11 21:36:38.06030693 +0000 UTC m=+1146.182638852"
Sep 11 21:36:38.579402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:38.579371    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 21:36:38.579603 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:38.579427    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 21:36:39.579879 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:39.579817    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 21:36:39.580219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:39.579905    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 21:36:40.579346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:40.579313    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 21:36:40.579521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:40.579377    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 21:36:41.582299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:41.582270    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:41.582681 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:41.582538    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:41.583165 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:41.583150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:36:45.893298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:45.893257    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:36:46.672691 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:46.672658    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:46.672977 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:46.672812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:46.685396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:46.685370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:47.116344 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:47.116316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:36:58.169147 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.169067    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:36:58.174719 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.174697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.177126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.177105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e0c211-serving-cert\""
Sep 11 21:36:58.184933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.184183    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:36:58.262364 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.262332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.262494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.262375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.262494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.262453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.262589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.262552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfstr\" (UniqueName: \"kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.363372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.363340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfstr\" (UniqueName: \"kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.363556 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.363423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.363556 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.363445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.363556 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.363486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.363691 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:58.363591    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-serving-cert: secret "maas-api-e2e-shared-b-w3-e0c211-serving-cert" not found
Sep 11 21:36:58.363691 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:36:58.363666    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls podName:ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b nodeName:}" failed. No retries permitted until 2026-09-11 21:36:58.863651233 +0000 UTC m=+1166.985983131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" (UID: "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b") : secret "maas-api-e2e-shared-b-w3-e0c211-serving-cert" not found
Sep 11 21:36:58.364102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.364074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.366164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.366138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.371888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.371865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfstr\" (UniqueName: \"kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.867522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.867460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:58.869911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:58.869883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:59.090553 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:59.090524    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:36:59.361188 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:36:59.361164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:36:59.363084 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:36:59.363057    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac31b6e4_acc0_41dd_b4ca_c8f5bb536a6b.slice/crio-2e2810fb56cc6a2d138068203919dec0031345845290bb63381d51f30f35c483 WatchSource:0}: Error finding container 2e2810fb56cc6a2d138068203919dec0031345845290bb63381d51f30f35c483: Status 404 returned error can't find the container with id 2e2810fb56cc6a2d138068203919dec0031345845290bb63381d51f30f35c483
Sep 11 21:37:00.182952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:00.182915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" event={"ID":"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b","Type":"ContainerStarted","Data":"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"}
Sep 11 21:37:00.183138 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:00.182979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" event={"ID":"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b","Type":"ContainerStarted","Data":"2e2810fb56cc6a2d138068203919dec0031345845290bb63381d51f30f35c483"}
Sep 11 21:37:00.183138 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:00.183008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:37:00.203234 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:00.203193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" podStartSLOduration=1.921001399 podStartE2EDuration="2.203177087s" totalImagesPullingTime="282.175688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:37:00.20177203 +0000 UTC m=+1168.324103950" watchObservedRunningTime="2026-09-11 21:37:00.203177087 +0000 UTC m=+1168.325509007"
Sep 11 21:37:06.193039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:06.193008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:37:23.790805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:23.790774    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:37:24.094299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.094212    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:37:24.098875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.098855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.105808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.105784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bc0b206e\"/\"kube-root-ca.crt\""
Sep 11 21:37:24.105932 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.105838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bc0b206e\"/\"openshift-service-ca.crt\""
Sep 11 21:37:24.105932 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.105839    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bc0b206e\"/\"e2e-lifecycle-model-bc0b206e-kserve-self-signed-certs\""
Sep 11 21:37:24.105932 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.105867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bc0b206e\"/\"default-dockercfg-snc76\""
Sep 11 21:37:24.118017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.117999    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:37:24.194926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.194898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.195070 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.194931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.195070 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.194954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.195070 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.195056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.195185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.195094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbz8w\" (UniqueName: \"kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.195185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.195137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbz8w\" (UniqueName: \"kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295854 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.295936 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.295901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.296031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.296011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.296095 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.296028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.296258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.296236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.298124 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.298108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.298402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.298383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.304869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.304847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbz8w\" (UniqueName: \"kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w\") pod \"e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") " pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.408328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.408245    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:24.692582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:24.692560    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:37:24.693830 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:37:24.693806    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c3d823b_46f1_404d_9a21_fee3ae2c27a3.slice/crio-98de25a84d3d4c049b942b516410eeab1a3b947c92a58177eb8220e17fa1937d WatchSource:0}: Error finding container 98de25a84d3d4c049b942b516410eeab1a3b947c92a58177eb8220e17fa1937d: Status 404 returned error can't find the container with id 98de25a84d3d4c049b942b516410eeab1a3b947c92a58177eb8220e17fa1937d
Sep 11 21:37:25.314911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:25.314870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" event={"ID":"1c3d823b-46f1-404d-9a21-fee3ae2c27a3","Type":"ContainerStarted","Data":"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"}
Sep 11 21:37:25.314911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:25.314914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" event={"ID":"1c3d823b-46f1-404d-9a21-fee3ae2c27a3","Type":"ContainerStarted","Data":"98de25a84d3d4c049b942b516410eeab1a3b947c92a58177eb8220e17fa1937d"}
Sep 11 21:37:25.337875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:25.337832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" podStartSLOduration=1.337817104 podStartE2EDuration="1.337817104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:37:25.337330772 +0000 UTC m=+1193.459662776" watchObservedRunningTime="2026-09-11 21:37:25.337817104 +0000 UTC m=+1193.460149023"
Sep 11 21:37:32.931617 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:32.931585    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:37:34.408738 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:34.408710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:34.408738 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:34.408745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:34.420665 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:34.420635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:35.385599 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:35.385567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:37:47.950885 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:47.950822    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" podUID="ce492243-3c20-486c-92a5-586f1dae9c4d" containerName="main" containerID="cri-o://75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5" gracePeriod=45
Sep 11 21:37:48.202270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.202215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:37:48.309113 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7649\" (UniqueName: \"kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309155    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs\") pod \"ce492243-3c20-486c-92a5-586f1dae9c4d\" (UID: \"ce492243-3c20-486c-92a5-586f1dae9c4d\") "
Sep 11 21:37:48.309404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309358    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:37:48.309634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:37:48.309634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home" (OuterVolumeSpecName: "home") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:37:48.309769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309749    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.309829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309774    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.309829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.309785    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.311447 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.311422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm" (OuterVolumeSpecName: "dshm") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:37:48.311579 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.311554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:37:48.311641 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.311595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649" (OuterVolumeSpecName: "kube-api-access-x7649") pod "ce492243-3c20-486c-92a5-586f1dae9c4d" (UID: "ce492243-3c20-486c-92a5-586f1dae9c4d"). InnerVolumeSpecName "kube-api-access-x7649". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:37:48.411177 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.411149    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7649\" (UniqueName: \"kubernetes.io/projected/ce492243-3c20-486c-92a5-586f1dae9c4d-kube-api-access-x7649\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.411177 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.411173    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce492243-3c20-486c-92a5-586f1dae9c4d-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.411330 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.411183    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce492243-3c20-486c-92a5-586f1dae9c4d-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:37:48.438272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.438243    2580 generic.go:356] "Generic (PLEG): container finished" podID="ce492243-3c20-486c-92a5-586f1dae9c4d" containerID="75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5" exitCode=0
Sep 11 21:37:48.438407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.438296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"
Sep 11 21:37:48.438407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.438338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" event={"ID":"ce492243-3c20-486c-92a5-586f1dae9c4d","Type":"ContainerDied","Data":"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"}
Sep 11 21:37:48.438407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.438399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp" event={"ID":"ce492243-3c20-486c-92a5-586f1dae9c4d","Type":"ContainerDied","Data":"1323759f9b74ec25444c78a65e06d35a71eac0c41dafa78cee16dcc8c3c33da3"}
Sep 11 21:37:48.438576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.438420    2580 scope.go:117] "RemoveContainer" containerID="75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"
Sep 11 21:37:48.447194 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.447179    2580 scope.go:117] "RemoveContainer" containerID="75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"
Sep 11 21:37:48.447473 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:37:48.447456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5\": container with ID starting with 75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5 not found: ID does not exist" containerID="75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"
Sep 11 21:37:48.447550 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.447481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5"} err="failed to get container status \"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5\": rpc error: code = NotFound desc = could not find container \"75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5\": container with ID starting with 75c55adcfde0def8065b2781aefc9c90eb709293b89e5779b79c7f5d702551f5 not found: ID does not exist"
Sep 11 21:37:48.455203 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.455157    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:37:48.459176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:48.459158    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/e2e-conflict-b-315f74cf-kserve-5476ccdc6d-kt5wp"]
Sep 11 21:37:50.094516 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:50.094472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:37:50.385660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:37:50.385581    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce492243-3c20-486c-92a5-586f1dae9c4d" path="/var/lib/kubelet/pods/ce492243-3c20-486c-92a5-586f1dae9c4d/volumes"
Sep 11 21:38:20.593104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.593072    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:38:20.593573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.593471    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce492243-3c20-486c-92a5-586f1dae9c4d" containerName="main"
Sep 11 21:38:20.593573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.593483    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce492243-3c20-486c-92a5-586f1dae9c4d" containerName="main"
Sep 11 21:38:20.593573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.593569    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce492243-3c20-486c-92a5-586f1dae9c4d" containerName="main"
Sep 11 21:38:20.597151 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.597122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.601212 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.601179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"distinct-w4-83ba87-kserve-self-signed-certs\""
Sep 11 21:38:20.611488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.611463    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:38:20.791493 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.791688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.791688 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.791808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lqs\" (UniqueName: \"kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.791808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.791914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.791839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lqs\" (UniqueName: \"kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892768 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892897 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.892958 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.892925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.894886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.894867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.895154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.895133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.900391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.900366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lqs\" (UniqueName: \"kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs\") pod \"distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:20.910691 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:20.910670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:21.184618 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:21.184592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:38:21.186113 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:38:21.186091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85549863_90e8_42b0_aaf6_e45573a66410.slice/crio-16425b32d7cb7aab1ba1b11f230f361b6ffb8b5fef9c0beded6e3a69870f9619 WatchSource:0}: Error finding container 16425b32d7cb7aab1ba1b11f230f361b6ffb8b5fef9c0beded6e3a69870f9619: Status 404 returned error can't find the container with id 16425b32d7cb7aab1ba1b11f230f361b6ffb8b5fef9c0beded6e3a69870f9619
Sep 11 21:38:21.404727 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:21.404690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:38:21.614005 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:21.613965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" event={"ID":"85549863-90e8-42b0-aaf6-e45573a66410","Type":"ContainerStarted","Data":"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"}
Sep 11 21:38:21.614005 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:21.614010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" event={"ID":"85549863-90e8-42b0-aaf6-e45573a66410","Type":"ContainerStarted","Data":"16425b32d7cb7aab1ba1b11f230f361b6ffb8b5fef9c0beded6e3a69870f9619"}
Sep 11 21:38:21.630648 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:21.630605    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" podStartSLOduration=1.630591374 podStartE2EDuration="1.630591374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:38:21.629101229 +0000 UTC m=+1249.751433148" watchObservedRunningTime="2026-09-11 21:38:21.630591374 +0000 UTC m=+1249.752923293"
Sep 11 21:38:30.911439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:30.911362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:30.911439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:30.911409    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:30.923619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:30.923593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:31.680423 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:31.680386    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:38:32.504978 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:32.504946    2580 scope.go:117] "RemoveContainer" containerID="457cb73b0081f719bb15828898c85a669bd8b53190588d75820197be6e08a750"
Sep 11 21:38:39.972601 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:39.972564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:38:54.802854 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:54.802824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:38:54.991840 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:54.991774    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" podUID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" containerName="main" containerID="cri-o://95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f" gracePeriod=45
Sep 11 21:38:55.244053 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.244030    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:38:55.396848 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.396848 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.397071 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.397071 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbz8w\" (UniqueName: \"kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.397071 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.397071 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.396978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm\") pod \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\" (UID: \"1c3d823b-46f1-404d-9a21-fee3ae2c27a3\") "
Sep 11 21:38:55.397071 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home" (OuterVolumeSpecName: "home") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:38:55.397294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:38:55.397294 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:38:55.397396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397359    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.397396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397371    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.397396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.397380    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.399287 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.399259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:38:55.399407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.399371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w" (OuterVolumeSpecName: "kube-api-access-hbz8w") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "kube-api-access-hbz8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:38:55.399491 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.399471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm" (OuterVolumeSpecName: "dshm") pod "1c3d823b-46f1-404d-9a21-fee3ae2c27a3" (UID: "1c3d823b-46f1-404d-9a21-fee3ae2c27a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:38:55.498331 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.498293    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbz8w\" (UniqueName: \"kubernetes.io/projected/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-kube-api-access-hbz8w\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.498331 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.498326    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.498331 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.498336    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3d823b-46f1-404d-9a21-fee3ae2c27a3-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:38:55.792823 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.792787    2580 generic.go:356] "Generic (PLEG): container finished" podID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" containerID="95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f" exitCode=0
Sep 11 21:38:55.793022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.792835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"
Sep 11 21:38:55.793022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.792873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" event={"ID":"1c3d823b-46f1-404d-9a21-fee3ae2c27a3","Type":"ContainerDied","Data":"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"}
Sep 11 21:38:55.793022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.792923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822" event={"ID":"1c3d823b-46f1-404d-9a21-fee3ae2c27a3","Type":"ContainerDied","Data":"98de25a84d3d4c049b942b516410eeab1a3b947c92a58177eb8220e17fa1937d"}
Sep 11 21:38:55.793022 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.792945    2580 scope.go:117] "RemoveContainer" containerID="95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"
Sep 11 21:38:55.802604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.802589    2580 scope.go:117] "RemoveContainer" containerID="95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"
Sep 11 21:38:55.802876 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:38:55.802859    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f\": container with ID starting with 95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f not found: ID does not exist" containerID="95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"
Sep 11 21:38:55.803126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.802888    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f"} err="failed to get container status \"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f\": rpc error: code = NotFound desc = could not find container \"95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f\": container with ID starting with 95e1540eea29dddb11f9107854e2533d18a2934f2d4003c2724aed7ed6a8d37f not found: ID does not exist"
Sep 11 21:38:55.813485 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.813463    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:38:55.820057 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:55.820036    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bc0b206e/e2e-lifecycle-model-bc0b206e-kserve-645698fdf9-lq822"]
Sep 11 21:38:56.389671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:38:56.389631    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" path="/var/lib/kubelet/pods/1c3d823b-46f1-404d-9a21-fee3ae2c27a3/volumes"
Sep 11 21:39:20.789923 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:20.789892    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:39:24.181876 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:24.181837    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:39:24.182321 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:24.182039    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" containerID="cri-o://94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619" gracePeriod=30
Sep 11 21:39:26.585097 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:26.585063    2580 patch_prober.go:28] interesting pod/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:39:26.585577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:26.585126    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:39:29.435805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.435784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:39:29.501261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501236    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501307    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-497nm\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501379 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data" (OuterVolumeSpecName: "istio-data") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:39:29.501604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501595    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo\") pod \"d424bf60-80ee-4c9a-85c7-680efd69a0be\" (UID: \"d424bf60-80ee-4c9a-85c7-680efd69a0be\") "
Sep 11 21:39:29.501862 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:39:29.501917 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:39:29.502309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.502047    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.502309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.502072    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d424bf60-80ee-4c9a-85c7-680efd69a0be-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.502309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.502087    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.502309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.501731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:39:29.502309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.502287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:39:29.503678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.503658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm" (OuterVolumeSpecName: "kube-api-access-497nm") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "kube-api-access-497nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:29.503752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.503715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token" (OuterVolumeSpecName: "istio-token") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:39:29.504217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.504192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:39:29.504217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.504205    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d424bf60-80ee-4c9a-85c7-680efd69a0be" (UID: "d424bf60-80ee-4c9a-85c7-680efd69a0be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:39:29.603268 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603241    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.603268 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603265    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.603268 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603275    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.603463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603283    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497nm\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-kube-api-access-497nm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.603463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603292    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d424bf60-80ee-4c9a-85c7-680efd69a0be-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.603463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.603300    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d424bf60-80ee-4c9a-85c7-680efd69a0be-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:39:29.976887 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.976802    2580 generic.go:356] "Generic (PLEG): container finished" podID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerID="94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619" exitCode=0
Sep 11 21:39:29.976887 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.976852    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"
Sep 11 21:39:29.977068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.976883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerDied","Data":"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"}
Sep 11 21:39:29.977068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.976936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl" event={"ID":"d424bf60-80ee-4c9a-85c7-680efd69a0be","Type":"ContainerDied","Data":"ee1fe2fbfdfefd8d73d845d779ef3edca49121f5b0db1c6b95da400ed1fa66c1"}
Sep 11 21:39:29.977068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.976957    2580 scope.go:117] "RemoveContainer" containerID="94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"
Sep 11 21:39:29.993128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:29.993096    2580 scope.go:117] "RemoveContainer" containerID="012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"
Sep 11 21:39:30.000909 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.000892    2580 scope.go:117] "RemoveContainer" containerID="94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"
Sep 11 21:39:30.001152 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:39:30.001137    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619\": container with ID starting with 94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619 not found: ID does not exist" containerID="94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"
Sep 11 21:39:30.001198 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.001158    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619"} err="failed to get container status \"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619\": rpc error: code = NotFound desc = could not find container \"94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619\": container with ID starting with 94611b18febb57bfe8d49eede8869b4747a1428682cf12f2169f234174dc7619 not found: ID does not exist"
Sep 11 21:39:30.001198 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.001173    2580 scope.go:117] "RemoveContainer" containerID="012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"
Sep 11 21:39:30.001411 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:39:30.001394    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4\": container with ID starting with 012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4 not found: ID does not exist" containerID="012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"
Sep 11 21:39:30.001454 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.001414    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4"} err="failed to get container status \"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4\": rpc error: code = NotFound desc = could not find container \"012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4\": container with ID starting with 012aab07d27fa9f5bce16313fe37e7e34c41ce4b80f786e2d252870d0c6a40c4 not found: ID does not exist"
Sep 11 21:39:30.005183 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.005160    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:39:30.009384 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.009364    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc0b206e-openshift-default-6b6df9c7cb-r5lpl"]
Sep 11 21:39:30.384156 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:30.384129    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" path="/var/lib/kubelet/pods/d424bf60-80ee-4c9a-85c7-680efd69a0be/volumes"
Sep 11 21:39:32.529291 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:32.529257    2580 scope.go:117] "RemoveContainer" containerID="c314128c2361f2550cfc25b0c5616a02f24d2f07ba84352475bcb59a32c37e4f"
Sep 11 21:39:43.384517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:39:43.384869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384834    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:39:43.384869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384846    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:39:43.384869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384856    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" containerName="main"
Sep 11 21:39:43.384869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384863    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" containerName="main"
Sep 11 21:39:43.384869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:39:43.385031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384874    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:39:43.385031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384933    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:39:43.385031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.384939    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3d823b-46f1-404d-9a21-fee3ae2c27a3" containerName="main"
Sep 11 21:39:43.389405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.389382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.391991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.391967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-83ba87\"/\"distinct-2-w4-83ba87-kserve-self-signed-certs\""
Sep 11 21:39:43.400347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.400328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:39:43.518879 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.518848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.519090 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.518888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.519090 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.518964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.519090 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.519062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.519594 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.519228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4khk\" (UniqueName: \"kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.519594 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.519481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620397 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620397 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4khk\" (UniqueName: \"kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620842 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620842 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.620917 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.620900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.622773 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.622752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.623104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.623088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.628844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.628812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4khk\" (UniqueName: \"kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk\") pod \"distinct-2-w4-83ba87-kserve-7d464d4b7c-92449\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") " pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.698710 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.698652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:43.992836 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:43.992814    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:39:43.994969 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:39:43.994945    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce9ba619_f092_4c3a_8348_51e69a961b59.slice/crio-c9ea3779c252116072f96c659e327ca03f02840f8085408043deb359f8ceb991 WatchSource:0}: Error finding container c9ea3779c252116072f96c659e327ca03f02840f8085408043deb359f8ceb991: Status 404 returned error can't find the container with id c9ea3779c252116072f96c659e327ca03f02840f8085408043deb359f8ceb991
Sep 11 21:39:44.051555 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:44.051532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" event={"ID":"ce9ba619-f092-4c3a-8348-51e69a961b59","Type":"ContainerStarted","Data":"c9ea3779c252116072f96c659e327ca03f02840f8085408043deb359f8ceb991"}
Sep 11 21:39:45.059263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:45.059227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" event={"ID":"ce9ba619-f092-4c3a-8348-51e69a961b59","Type":"ContainerStarted","Data":"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"}
Sep 11 21:39:53.493390 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:53.493287    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" podStartSLOduration=10.493271524 podStartE2EDuration="10.493271524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:45.075743102 +0000 UTC m=+1333.198075021" watchObservedRunningTime="2026-09-11 21:39:53.493271524 +0000 UTC m=+1341.615603443"
Sep 11 21:39:53.495262 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:53.495243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:39:53.699154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:53.699120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:53.699309 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:53.699173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:53.711959 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:53.711937    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:39:54.114494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:39:54.114463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:40:18.600374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.600338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:40:18.777285 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.777252    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:40:18.777914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.777897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d424bf60-80ee-4c9a-85c7-680efd69a0be" containerName="istio-proxy"
Sep 11 21:40:18.787574 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.787546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.790820 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.790791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-78e0f8-openshift-default-dockercfg-kb8lp\""
Sep 11 21:40:18.794478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.794458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:40:18.807521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.807947 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.807803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzd2\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909118 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzd2\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909290 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909796 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909858 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.909996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.910119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.909977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.910519 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.910472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.912626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.912605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.912865 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.912847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.919058 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.919038    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:18.919280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:18.919259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzd2\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2\") pod \"e2e-api-a-78e0f8-openshift-default-65f487984-8bqns\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") " pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:19.102385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:19.102356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:19.381655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:19.381627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:40:19.383385 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:40:19.383350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740c22fe_f3cf_4495_9402_153516b09a19.slice/crio-5bc2b27e16033ee40bfa22d2d954e90519a48c64fc5a0b8f82bf44394e48bbee WatchSource:0}: Error finding container 5bc2b27e16033ee40bfa22d2d954e90519a48c64fc5a0b8f82bf44394e48bbee: Status 404 returned error can't find the container with id 5bc2b27e16033ee40bfa22d2d954e90519a48c64fc5a0b8f82bf44394e48bbee
Sep 11 21:40:19.385966 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:19.385924    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:19.386049 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:19.386013    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:19.386088 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:19.386054    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:20.244283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:20.244251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerStarted","Data":"a61ae5de6730bdfefe799cb8d8dbbc38ab3a387281c112aa0f8b844b8cb4ed7d"}
Sep 11 21:40:20.244671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:20.244303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerStarted","Data":"5bc2b27e16033ee40bfa22d2d954e90519a48c64fc5a0b8f82bf44394e48bbee"}
Sep 11 21:40:20.262517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:20.262465    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podStartSLOduration=2.262452149 podStartE2EDuration="2.262452149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:40:20.261581984 +0000 UTC m=+1368.383913906" watchObservedRunningTime="2026-09-11 21:40:20.262452149 +0000 UTC m=+1368.384784068"
Sep 11 21:40:21.103136 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:21.103101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:21.108191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:21.108169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:21.250337 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:21.250309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:21.251193 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:21.251175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:40:48.295597 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:48.295563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:40:53.099453 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.099426    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:40:53.104289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.104267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.106643 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.106623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d2170a-openshift-default-dockercfg-x2swf\""
Sep 11 21:40:53.118611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.118591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:40:53.232079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.232040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.232381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.232360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.232573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.232555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.232789 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.232774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.232926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.232910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.233056 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.233040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptk4z\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.233206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.233192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.233349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.233336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.233517 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.233483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.334856 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.334824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.335141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.335110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.335289 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.335272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptk4z\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.335482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.335466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.335793 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.335664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.336272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.336251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.336456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.336431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.336614 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.336596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.336763 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.336748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.337859 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.337392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.340216 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.338770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.340216 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.339931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.340216 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.340168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.340697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.340661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.341125 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.341099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.348081 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.348061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.349697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.349643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.349827 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.349766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptk4z\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z\") pod \"e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") " pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.414240 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.414215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:53.738577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.738542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:40:53.744864 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:40:53.744832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c97730_7b30_4ec2_8822_f521196470fb.slice/crio-49cc14c8ac43993a2d2807f4d9f6d2fb98802fd9dbc698079e146253b263e1f1 WatchSource:0}: Error finding container 49cc14c8ac43993a2d2807f4d9f6d2fb98802fd9dbc698079e146253b263e1f1: Status 404 returned error can't find the container with id 49cc14c8ac43993a2d2807f4d9f6d2fb98802fd9dbc698079e146253b263e1f1
Sep 11 21:40:53.750679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.748381    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:53.750679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.748465    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:53.750679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:53.748529    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:54.426577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:54.426547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerStarted","Data":"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"}
Sep 11 21:40:54.426941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:54.426589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerStarted","Data":"49cc14c8ac43993a2d2807f4d9f6d2fb98802fd9dbc698079e146253b263e1f1"}
Sep 11 21:40:54.475652 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:54.475591    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podStartSLOduration=1.475574498 podStartE2EDuration="1.475574498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:40:54.471721022 +0000 UTC m=+1402.594052941" watchObservedRunningTime="2026-09-11 21:40:54.475574498 +0000 UTC m=+1402.597906418"
Sep 11 21:40:55.415198 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:55.415165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:55.421371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:55.421345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:55.432741 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:55.432709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:55.433552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:55.433535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:40:58.451832 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.451806    2580 generic.go:356] "Generic (PLEG): container finished" podID="740c22fe-f3cf-4495-9402-153516b09a19" containerID="a61ae5de6730bdfefe799cb8d8dbbc38ab3a387281c112aa0f8b844b8cb4ed7d" exitCode=0
Sep 11 21:40:58.452157 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.451874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerDied","Data":"a61ae5de6730bdfefe799cb8d8dbbc38ab3a387281c112aa0f8b844b8cb4ed7d"}
Sep 11 21:40:58.452293 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.452279    2580 scope.go:117] "RemoveContainer" containerID="a61ae5de6730bdfefe799cb8d8dbbc38ab3a387281c112aa0f8b844b8cb4ed7d"
Sep 11 21:40:58.453399 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.453369    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:58.453474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.453439    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:58.453474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:58.453470    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:40:59.460331 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:40:59.460288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerStarted","Data":"ba343f1ec4d83f5c687f0bfe16aa94a0d1f18319e2b42f87f97d400e71d9b158"}
Sep 11 21:41:00.103581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:00.103545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:00.103775 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:00.103753    2580 patch_prober.go:28] interesting pod/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 21:41:00.103868 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:00.103805    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 21:41:01.103412 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.103370    2580 patch_prober.go:28] interesting pod/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 21:41:01.103892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.103433    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 21:41:01.861822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.861783    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:41:01.866254 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.866228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:01.868690 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.868666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d2170a-serving-cert\""
Sep 11 21:41:01.876409 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.876385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:41:01.949098 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.949062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:01.949333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.949313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:01.949457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.949442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clldl\" (UniqueName: \"kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:01.949603 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:01.949586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.050998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.050950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.051194 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.051006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clldl\" (UniqueName: \"kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.051194 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.051041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.051343 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.051218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.051403 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:41:02.051362    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-serving-cert: secret "maas-api-e2e-api-b-d2170a-serving-cert" not found
Sep 11 21:41:02.051473 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:41:02.051456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls podName:97477337-3245-4fed-9eea-0a5ae1103e74 nodeName:}" failed. No retries permitted until 2026-09-11 21:41:02.551434991 +0000 UTC m=+1410.673766895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls") pod "maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" (UID: "97477337-3245-4fed-9eea-0a5ae1103e74") : secret "maas-api-e2e-api-b-d2170a-serving-cert" not found
Sep 11 21:41:02.052735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.052711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.056826 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.056777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.060195 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.060173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clldl\" (UniqueName: \"kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.103732 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.103703    2580 patch_prober.go:28] interesting pod/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 21:41:02.104047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.103763    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 21:41:02.559671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.559632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.562141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.562113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") pod \"maas-api-e2e-api-b-d2170a-85d9896895-k9lg4\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:02.779626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:02.779593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:03.075607 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:03.075580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:41:03.077361 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:41:03.077310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97477337_3245_4fed_9eea_0a5ae1103e74.slice/crio-58adeab6ccfe06a2844f421f8ff787ba963c6d2d5684504e0d4d2c2c8e11b544 WatchSource:0}: Error finding container 58adeab6ccfe06a2844f421f8ff787ba963c6d2d5684504e0d4d2c2c8e11b544: Status 404 returned error can't find the container with id 58adeab6ccfe06a2844f421f8ff787ba963c6d2d5684504e0d4d2c2c8e11b544
Sep 11 21:41:03.078720 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:03.078702    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:41:03.103758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:03.103731    2580 patch_prober.go:28] interesting pod/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 21:41:03.104104 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:03.103794    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 21:41:03.482332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:03.482263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" event={"ID":"97477337-3245-4fed-9eea-0a5ae1103e74","Type":"ContainerStarted","Data":"58adeab6ccfe06a2844f421f8ff787ba963c6d2d5684504e0d4d2c2c8e11b544"}
Sep 11 21:41:04.103310 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:04.103272    2580 patch_prober.go:28] interesting pod/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 21:41:04.103478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:04.103333    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 21:41:04.488882 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:04.488846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" event={"ID":"97477337-3245-4fed-9eea-0a5ae1103e74","Type":"ContainerStarted","Data":"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"}
Sep 11 21:41:04.489247 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:04.488980    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:04.509565 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:04.509482    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" podStartSLOduration=3.092482657 podStartE2EDuration="3.509464819s" totalImagesPullingTime="416.982162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:41:04.508924083 +0000 UTC m=+1412.631256003" watchObservedRunningTime="2026-09-11 21:41:04.509464819 +0000 UTC m=+1412.631796741"
Sep 11 21:41:05.106913 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:05.106885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:05.107238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:05.107219    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:05.107951 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:05.107933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:10.499371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:10.499345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:41:24.600733 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.600669    2580 generic.go:356] "Generic (PLEG): container finished" podID="e3c97730-7b30-4ec2-8822-f521196470fb" containerID="3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc" exitCode=0
Sep 11 21:41:24.601092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.600733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerDied","Data":"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"}
Sep 11 21:41:24.601092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.601047    2580 scope.go:117] "RemoveContainer" containerID="3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"
Sep 11 21:41:24.602258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.602217    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:41:24.602357 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.602312    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:41:24.602424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:24.602371    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:41:25.433119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:25.433087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:41:25.608391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:25.608361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerStarted","Data":"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"}
Sep 11 21:41:26.414572 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:26.414542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:41:26.415651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:26.415627    2580 patch_prober.go:28] interesting pod/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 11 21:41:26.415751 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:26.415687    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 11 21:41:27.415017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:27.414983    2580 patch_prober.go:28] interesting pod/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 11 21:41:27.415386 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:27.415052    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 11 21:41:28.415225 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:28.415191    2580 patch_prober.go:28] interesting pod/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 11 21:41:28.415622 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:28.415253    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 11 21:41:29.414604 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:29.414573    2580 patch_prober.go:28] interesting pod/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 11 21:41:29.414781 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:29.414637    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 11 21:41:30.414732 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:30.414701    2580 patch_prober.go:28] interesting pod/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 11 21:41:30.415115 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:30.414750    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 11 21:41:31.418609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:31.418580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:41:31.418999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:31.418835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:41:31.419590 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:31.419573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:41:40.892875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:40.892841    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:41:52.619207 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:52.619173    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:41:52.619711 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:52.619422    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy" containerID="cri-o://ba343f1ec4d83f5c687f0bfe16aa94a0d1f18319e2b42f87f97d400e71d9b158" gracePeriod=30
Sep 11 21:41:54.266140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:54.266108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:41:54.271375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:54.271349    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:41:54.293373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:54.293329    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:41:54.304096 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:54.304071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:41:57.791453 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.791411    2580 generic.go:356] "Generic (PLEG): container finished" podID="740c22fe-f3cf-4495-9402-153516b09a19" containerID="ba343f1ec4d83f5c687f0bfe16aa94a0d1f18319e2b42f87f97d400e71d9b158" exitCode=0
Sep 11 21:41:57.791839 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.791488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerDied","Data":"ba343f1ec4d83f5c687f0bfe16aa94a0d1f18319e2b42f87f97d400e71d9b158"}
Sep 11 21:41:57.791839 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.791579    2580 scope.go:117] "RemoveContainer" containerID="a61ae5de6730bdfefe799cb8d8dbbc38ab3a387281c112aa0f8b844b8cb4ed7d"
Sep 11 21:41:57.872093 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.872066    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:57.993369 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993344    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzzd2\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993677    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.993907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993807    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:41:57.993907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993834    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:41:57.993907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo\") pod \"740c22fe-f3cf-4495-9402-153516b09a19\" (UID: \"740c22fe-f3cf-4495-9402-153516b09a19\") "
Sep 11 21:41:57.994060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.993931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:41:57.994060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:41:57.994157 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data" (OuterVolumeSpecName: "istio-data") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:41:57.994272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994254    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:57.994398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994293    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:57.994398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994307    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740c22fe-f3cf-4495-9402-153516b09a19-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:57.994398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994320    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:57.994398 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.994331    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:57.995728 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.995703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2" (OuterVolumeSpecName: "kube-api-access-hzzd2") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "kube-api-access-hzzd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:41:57.996297 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.996275    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:41:57.996391 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.996375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:41:57.996463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:57.996449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token" (OuterVolumeSpecName: "istio-token") pod "740c22fe-f3cf-4495-9402-153516b09a19" (UID: "740c22fe-f3cf-4495-9402-153516b09a19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:41:58.095440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.095412    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740c22fe-f3cf-4495-9402-153516b09a19-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:58.095440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.095438    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740c22fe-f3cf-4495-9402-153516b09a19-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:58.095602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.095452    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzd2\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-kube-api-access-hzzd2\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:58.095602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.095464    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740c22fe-f3cf-4495-9402-153516b09a19-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:41:58.797374 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.797350    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"
Sep 11 21:41:58.797777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.797382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns" event={"ID":"740c22fe-f3cf-4495-9402-153516b09a19","Type":"ContainerDied","Data":"5bc2b27e16033ee40bfa22d2d954e90519a48c64fc5a0b8f82bf44394e48bbee"}
Sep 11 21:41:58.797777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.797436    2580 scope.go:117] "RemoveContainer" containerID="ba343f1ec4d83f5c687f0bfe16aa94a0d1f18319e2b42f87f97d400e71d9b158"
Sep 11 21:41:58.815612 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.815591    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:41:58.822836 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:41:58.822818    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-78e0f8-openshift-default-65f487984-8bqns"]
Sep 11 21:42:00.383991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:00.383958    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740c22fe-f3cf-4495-9402-153516b09a19" path="/var/lib/kubelet/pods/740c22fe-f3cf-4495-9402-153516b09a19/volumes"
Sep 11 21:42:05.601936 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:05.601904    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"]
Sep 11 21:42:05.605540 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:05.605515    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-db06d6-29819370-vmjx2"]
Sep 11 21:42:06.384064 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:06.384036    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15" path="/var/lib/kubelet/pods/2c5be0be-7ac8-4c69-9b5d-088f7aa2ef15/volumes"
Sep 11 21:42:09.285868 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.285830    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" podUID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" containerName="main" containerID="cri-o://9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c" gracePeriod=45
Sep 11 21:42:09.291082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.291029    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" podUID="2992f556-45ab-4435-8b97-940dc68e803e" containerName="main" containerID="cri-o://5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a" gracePeriod=45
Sep 11 21:42:09.312257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.312229    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" podUID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" containerName="main" containerID="cri-o://9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554" gracePeriod=45
Sep 11 21:42:09.322546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.322515    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" podUID="9ed9750a-a332-42d8-99f9-5195a692470a" containerName="main" containerID="cri-o://d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4" gracePeriod=45
Sep 11 21:42:09.630153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.630114    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:42:09.634049 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.634025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:42:09.640580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.640559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:42:09.653284 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.653268    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:42:09.700673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.700673 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700667    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700690    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkjjb\" (UniqueName: \"kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.700900 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700923    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700966    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dck5k\" (UniqueName: \"kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache\") pod \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\" (UID: \"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.701236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache\") pod \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\" (UID: \"772ef7ba-034e-446d-ac99-b0f4f5190d7e\") "
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jljrc\" (UniqueName: \"kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home\") pod \"2992f556-45ab-4435-8b97-940dc68e803e\" (UID: \"2992f556-45ab-4435-8b97-940dc68e803e\") "
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.700923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home" (OuterVolumeSpecName: "home") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home" (OuterVolumeSpecName: "home") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.701974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache" (OuterVolumeSpecName: "model-cache") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.702030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home" (OuterVolumeSpecName: "home") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.702438 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.702101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.703060 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.702656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.704877 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.704841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.706370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.706412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k" (OuterVolumeSpecName: "kube-api-access-dck5k") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "kube-api-access-dck5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.706584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm" (OuterVolumeSpecName: "dshm") pod "772ef7ba-034e-446d-ac99-b0f4f5190d7e" (UID: "772ef7ba-034e-446d-ac99-b0f4f5190d7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.706634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc" (OuterVolumeSpecName: "kube-api-access-jljrc") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "kube-api-access-jljrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.706720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm" (OuterVolumeSpecName: "dshm") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.707306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.707047    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm" (OuterVolumeSpecName: "dshm") pod "2992f556-45ab-4435-8b97-940dc68e803e" (UID: "2992f556-45ab-4435-8b97-940dc68e803e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.707730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.707704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb" (OuterVolumeSpecName: "kube-api-access-hkjjb") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "kube-api-access-hkjjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:09.708013 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.707980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" (UID: "f227fd7c-24a0-4e71-93b7-4e4bc0f3b646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:09.802917 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.802887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.802917 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.802921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.803140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.802976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.803140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.803140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.803140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache\") pod \"9ed9750a-a332-42d8-99f9-5195a692470a\" (UID: \"9ed9750a-a332-42d8-99f9-5195a692470a\") "
Sep 11 21:42:09.803347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.803347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home" (OuterVolumeSpecName: "home") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.803436 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.803535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803519    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803538    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803547    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803554    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljrc\" (UniqueName: \"kubernetes.io/projected/2992f556-45ab-4435-8b97-940dc68e803e-kube-api-access-jljrc\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803562    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803569    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803577    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803589    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkjjb\" (UniqueName: \"kubernetes.io/projected/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-kube-api-access-hkjjb\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803599    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803609    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803616    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803627    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803637    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803644    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803651    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2992f556-45ab-4435-8b97-940dc68e803e-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803659    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2992f556-45ab-4435-8b97-940dc68e803e-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803669    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772ef7ba-034e-446d-ac99-b0f4f5190d7e-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803681    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803693    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dck5k\" (UniqueName: \"kubernetes.io/projected/772ef7ba-034e-446d-ac99-b0f4f5190d7e-kube-api-access-dck5k\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803703    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772ef7ba-034e-446d-ac99-b0f4f5190d7e-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.803784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.803711    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.805065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.805046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm" (OuterVolumeSpecName: "dshm") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:09.805581 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.805565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p" (OuterVolumeSpecName: "kube-api-access-l6k8p") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "kube-api-access-l6k8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:09.805631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.805576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ed9750a-a332-42d8-99f9-5195a692470a" (UID: "9ed9750a-a332-42d8-99f9-5195a692470a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:09.861537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.861439    2580 generic.go:356] "Generic (PLEG): container finished" podID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" containerID="9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c" exitCode=0
Sep 11 21:42:09.861666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.861538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" event={"ID":"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646","Type":"ContainerDied","Data":"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"}
Sep 11 21:42:09.861666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.861561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"
Sep 11 21:42:09.861666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.861582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh" event={"ID":"f227fd7c-24a0-4e71-93b7-4e4bc0f3b646","Type":"ContainerDied","Data":"9837a099eefde3719ae0b10f2d47faeccb459dfcc702bf08f19aa6a39b90d92b"}
Sep 11 21:42:09.861666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.861604    2580 scope.go:117] "RemoveContainer" containerID="9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"
Sep 11 21:42:09.869038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.869000    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ed9750a-a332-42d8-99f9-5195a692470a" containerID="d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4" exitCode=0
Sep 11 21:42:09.869655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.869626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" event={"ID":"9ed9750a-a332-42d8-99f9-5195a692470a","Type":"ContainerDied","Data":"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"}
Sep 11 21:42:09.869757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.869699    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"
Sep 11 21:42:09.869757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.869675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d" event={"ID":"9ed9750a-a332-42d8-99f9-5195a692470a","Type":"ContainerDied","Data":"7385bc2335b93330e7b0202bc162c4ffc292700aef96e895545dd23e2c154227"}
Sep 11 21:42:09.872001 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.871978    2580 generic.go:356] "Generic (PLEG): container finished" podID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" containerID="9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554" exitCode=0
Sep 11 21:42:09.872171 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.872020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" event={"ID":"772ef7ba-034e-446d-ac99-b0f4f5190d7e","Type":"ContainerDied","Data":"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"}
Sep 11 21:42:09.872285 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.872175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8" event={"ID":"772ef7ba-034e-446d-ac99-b0f4f5190d7e","Type":"ContainerDied","Data":"6ace35eaddfe7df2a56ce1129928b35d8eb6b1d012feff88ae4af8c49d60b9a0"}
Sep 11 21:42:09.872285 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.872033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"
Sep 11 21:42:09.873852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.873832    2580 generic.go:356] "Generic (PLEG): container finished" podID="2992f556-45ab-4435-8b97-940dc68e803e" containerID="5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a" exitCode=0
Sep 11 21:42:09.873989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.873890    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"
Sep 11 21:42:09.874206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.873906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" event={"ID":"2992f556-45ab-4435-8b97-940dc68e803e","Type":"ContainerDied","Data":"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"}
Sep 11 21:42:09.874206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.874186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw" event={"ID":"2992f556-45ab-4435-8b97-940dc68e803e","Type":"ContainerDied","Data":"24b3ea5fcdf1955632b134f2d2769c0aea0b4d4fd2b70d92890e93ccabde76fe"}
Sep 11 21:42:09.881418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.881400    2580 scope.go:117] "RemoveContainer" containerID="9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"
Sep 11 21:42:09.881750 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:09.881729    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c\": container with ID starting with 9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c not found: ID does not exist" containerID="9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"
Sep 11 21:42:09.881833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.881762    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c"} err="failed to get container status \"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c\": rpc error: code = NotFound desc = could not find container \"9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c\": container with ID starting with 9417ad843d8f1555690a6de74935e7acb46d060eb0a997b8a14790f50c673c7c not found: ID does not exist"
Sep 11 21:42:09.881833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.881784    2580 scope.go:117] "RemoveContainer" containerID="d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"
Sep 11 21:42:09.891784 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.891769    2580 scope.go:117] "RemoveContainer" containerID="d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"
Sep 11 21:42:09.892017 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:09.891999    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4\": container with ID starting with d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4 not found: ID does not exist" containerID="d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"
Sep 11 21:42:09.892084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.892023    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4"} err="failed to get container status \"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4\": rpc error: code = NotFound desc = could not find container \"d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4\": container with ID starting with d56113aa1a6a6d1c1544f743c0797f1a21b27794fe89e45d7a808e8cbc8d91f4 not found: ID does not exist"
Sep 11 21:42:09.892084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.892038    2580 scope.go:117] "RemoveContainer" containerID="9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"
Sep 11 21:42:09.898957 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.898937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:42:09.900979 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.900961    2580 scope.go:117] "RemoveContainer" containerID="9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"
Sep 11 21:42:09.901255 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:09.901235    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554\": container with ID starting with 9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554 not found: ID does not exist" containerID="9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"
Sep 11 21:42:09.901332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.901259    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554"} err="failed to get container status \"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554\": rpc error: code = NotFound desc = could not find container \"9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554\": container with ID starting with 9c25add3096582c9e687d561c5e72b63e9f26d4db62e3b4a22848ba3d4cee554 not found: ID does not exist"
Sep 11 21:42:09.901332 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.901275    2580 scope.go:117] "RemoveContainer" containerID="5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"
Sep 11 21:42:09.901855 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.901833    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-2-w0-db06d6-kserve-bf786876b-k6qjh"]
Sep 11 21:42:09.904119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.904099    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ed9750a-a332-42d8-99f9-5195a692470a-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.904119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.904120    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ed9750a-a332-42d8-99f9-5195a692470a-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.904227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.904128    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/9ed9750a-a332-42d8-99f9-5195a692470a-kube-api-access-l6k8p\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:09.910149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.910126    2580 scope.go:117] "RemoveContainer" containerID="5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"
Sep 11 21:42:09.910413 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:09.910390    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a\": container with ID starting with 5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a not found: ID does not exist" containerID="5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"
Sep 11 21:42:09.910483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.910421    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a"} err="failed to get container status \"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a\": rpc error: code = NotFound desc = could not find container \"5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a\": container with ID starting with 5895fc2b91d5bfd3ba76d7145da19513dfb01c20773b534c8106b0923c9b089a not found: ID does not exist"
Sep 11 21:42:09.914677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.914654    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:42:09.916324 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.916303    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/premium-w0-db06d6-kserve-7fd6d46578-kwxq8"]
Sep 11 21:42:09.926995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.926974    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:42:09.945220 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.934653    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/simulator-w0-db06d6-kserve-59698c6b5f-4zz4d"]
Sep 11 21:42:09.953639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.953618    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:42:09.958141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:09.958121    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-db06d6/distinct-w0-db06d6-kserve-86b45fdb8f-npvcw"]
Sep 11 21:42:10.384942 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:10.384914    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2992f556-45ab-4435-8b97-940dc68e803e" path="/var/lib/kubelet/pods/2992f556-45ab-4435-8b97-940dc68e803e/volumes"
Sep 11 21:42:10.385300 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:10.385287    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" path="/var/lib/kubelet/pods/772ef7ba-034e-446d-ac99-b0f4f5190d7e/volumes"
Sep 11 21:42:10.385642 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:10.385630    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed9750a-a332-42d8-99f9-5195a692470a" path="/var/lib/kubelet/pods/9ed9750a-a332-42d8-99f9-5195a692470a/volumes"
Sep 11 21:42:10.386265 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:10.386252    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" path="/var/lib/kubelet/pods/f227fd7c-24a0-4e71-93b7-4e4bc0f3b646/volumes"
Sep 11 21:42:11.286514 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.286469    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:42:11.286753 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.286728    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" podUID="97477337-3245-4fed-9eea-0a5ae1103e74" containerName="maas-api" containerID="cri-o://f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7" gracePeriod=30
Sep 11 21:42:11.535958 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.535935    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:42:11.620276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.620212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs\") pod \"97477337-3245-4fed-9eea-0a5ae1103e74\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") "
Sep 11 21:42:11.620276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.620264    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clldl\" (UniqueName: \"kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl\") pod \"97477337-3245-4fed-9eea-0a5ae1103e74\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") "
Sep 11 21:42:11.620421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.620320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") pod \"97477337-3245-4fed-9eea-0a5ae1103e74\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") "
Sep 11 21:42:11.620421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.620340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca\") pod \"97477337-3245-4fed-9eea-0a5ae1103e74\" (UID: \"97477337-3245-4fed-9eea-0a5ae1103e74\") "
Sep 11 21:42:11.620737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.620707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97477337-3245-4fed-9eea-0a5ae1103e74" (UID: "97477337-3245-4fed-9eea-0a5ae1103e74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:42:11.622439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.622407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97477337-3245-4fed-9eea-0a5ae1103e74" (UID: "97477337-3245-4fed-9eea-0a5ae1103e74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:11.622552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.622489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97477337-3245-4fed-9eea-0a5ae1103e74" (UID: "97477337-3245-4fed-9eea-0a5ae1103e74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:11.622552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.622524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl" (OuterVolumeSpecName: "kube-api-access-clldl") pod "97477337-3245-4fed-9eea-0a5ae1103e74" (UID: "97477337-3245-4fed-9eea-0a5ae1103e74"). InnerVolumeSpecName "kube-api-access-clldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:11.721632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.721598    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clldl\" (UniqueName: \"kubernetes.io/projected/97477337-3245-4fed-9eea-0a5ae1103e74-kube-api-access-clldl\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:11.721632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.721627    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:11.721632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.721636    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97477337-3245-4fed-9eea-0a5ae1103e74-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:11.721632 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.721643    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97477337-3245-4fed-9eea-0a5ae1103e74-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:11.889018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.888934    2580 generic.go:356] "Generic (PLEG): container finished" podID="97477337-3245-4fed-9eea-0a5ae1103e74" containerID="f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7" exitCode=0
Sep 11 21:42:11.889018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.888983    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"
Sep 11 21:42:11.889219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.889020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" event={"ID":"97477337-3245-4fed-9eea-0a5ae1103e74","Type":"ContainerDied","Data":"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"}
Sep 11 21:42:11.889219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.889069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4" event={"ID":"97477337-3245-4fed-9eea-0a5ae1103e74","Type":"ContainerDied","Data":"58adeab6ccfe06a2844f421f8ff787ba963c6d2d5684504e0d4d2c2c8e11b544"}
Sep 11 21:42:11.889219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.889092    2580 scope.go:117] "RemoveContainer" containerID="f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"
Sep 11 21:42:11.898200 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.898185    2580 scope.go:117] "RemoveContainer" containerID="f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"
Sep 11 21:42:11.898448 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:11.898432    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7\": container with ID starting with f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7 not found: ID does not exist" containerID="f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"
Sep 11 21:42:11.898489 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.898456    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7"} err="failed to get container status \"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7\": rpc error: code = NotFound desc = could not find container \"f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7\": container with ID starting with f7efa87acc38e226246265ee568edc2be31ac2da0950d3beeaf5fa9cb1236bc7 not found: ID does not exist"
Sep 11 21:42:11.912999 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.912976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:42:11.916034 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:11.916014    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d2170a-85d9896895-k9lg4"]
Sep 11 21:42:12.384416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:12.384384    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97477337-3245-4fed-9eea-0a5ae1103e74" path="/var/lib/kubelet/pods/97477337-3245-4fed-9eea-0a5ae1103e74/volumes"
Sep 11 21:42:18.208101 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:18.208066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:42:18.208592 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:18.208317    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy" containerID="cri-o://1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3" gracePeriod=30
Sep 11 21:42:23.000036 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.000005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:42:23.468828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.468804    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:42:23.542949 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.542923    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.542962    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543030    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptk4z\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543092 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo\") pod \"e3c97730-7b30-4ec2-8822-f521196470fb\" (UID: \"e3c97730-7b30-4ec2-8822-f521196470fb\") "
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:23.543323 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543626    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543647    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543662    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543674    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.543683 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.543685    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c97730-7b30-4ec2-8822-f521196470fb-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.545269 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.545245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:23.545365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.545350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:42:23.545484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.545453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z" (OuterVolumeSpecName: "kube-api-access-ptk4z") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "kube-api-access-ptk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:23.545591 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.545529    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3c97730-7b30-4ec2-8822-f521196470fb" (UID: "e3c97730-7b30-4ec2-8822-f521196470fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:23.644350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.644291    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.644350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.644314    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c97730-7b30-4ec2-8822-f521196470fb-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.644350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.644323    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c97730-7b30-4ec2-8822-f521196470fb-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.644350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.644332    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptk4z\" (UniqueName: \"kubernetes.io/projected/e3c97730-7b30-4ec2-8822-f521196470fb-kube-api-access-ptk4z\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:23.960983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.960900    2580 generic.go:356] "Generic (PLEG): container finished" podID="e3c97730-7b30-4ec2-8822-f521196470fb" containerID="1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3" exitCode=0
Sep 11 21:42:23.960983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.960940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerDied","Data":"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"}
Sep 11 21:42:23.961154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.960992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm" event={"ID":"e3c97730-7b30-4ec2-8822-f521196470fb","Type":"ContainerDied","Data":"49cc14c8ac43993a2d2807f4d9f6d2fb98802fd9dbc698079e146253b263e1f1"}
Sep 11 21:42:23.961154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.961015    2580 scope.go:117] "RemoveContainer" containerID="1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"
Sep 11 21:42:23.961154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.960947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"
Sep 11 21:42:23.970957 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.970940    2580 scope.go:117] "RemoveContainer" containerID="3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"
Sep 11 21:42:23.979219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.979204    2580 scope.go:117] "RemoveContainer" containerID="1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"
Sep 11 21:42:23.979486 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:23.979468    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3\": container with ID starting with 1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3 not found: ID does not exist" containerID="1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"
Sep 11 21:42:23.979551 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.979495    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3"} err="failed to get container status \"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3\": rpc error: code = NotFound desc = could not find container \"1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3\": container with ID starting with 1d5a495b9d3538b871ce05f9580a89fa17f5767e52ee113c7d2e125bbbcc44f3 not found: ID does not exist"
Sep 11 21:42:23.979551 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.979541    2580 scope.go:117] "RemoveContainer" containerID="3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"
Sep 11 21:42:23.979816 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:23.979799    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc\": container with ID starting with 3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc not found: ID does not exist" containerID="3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"
Sep 11 21:42:23.979875 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.979822    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc"} err="failed to get container status \"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc\": rpc error: code = NotFound desc = could not find container \"3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc\": container with ID starting with 3ecd43f4bdfc8e2d7429e40f37bf7c01448b416ef4ec66856216a4e7ed93e1fc not found: ID does not exist"
Sep 11 21:42:23.985322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.985297    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:42:23.989922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:23.989902    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d2170a-openshift-default-6b87b7d64d-8p5vm"]
Sep 11 21:42:24.387933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:24.387898    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" path="/var/lib/kubelet/pods/e3c97730-7b30-4ec2-8822-f521196470fb/volumes"
Sep 11 21:42:32.572114 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:32.572084    2580 scope.go:117] "RemoveContainer" containerID="26078ba125765f31b8eb7f3a94dfaa7e9f0506e7a85e9185bbde21bac4bb5c5b"
Sep 11 21:42:32.582190 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:32.582167    2580 scope.go:117] "RemoveContainer" containerID="94b01f19c9440ae119fe730f36e8ee70b30111219f4b40684c0cdcdfbefa7527"
Sep 11 21:42:35.023328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:35.023295    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:42:35.030708 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:35.030680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:42:35.041515 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:35.041459    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:42:35.054266 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:35.054238    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:42:50.043585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.043543    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" podUID="ce9ba619-f092-4c3a-8348-51e69a961b59" containerName="main" containerID="cri-o://b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817" gracePeriod=45
Sep 11 21:42:50.049280 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.049232    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" podUID="85549863-90e8-42b0-aaf6-e45573a66410" containerName="main" containerID="cri-o://94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce" gracePeriod=45
Sep 11 21:42:50.064925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.064883    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" podUID="edd3c8af-36ff-40fe-be3c-13aa48298e39" containerName="main" containerID="cri-o://0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40" gracePeriod=45
Sep 11 21:42:50.071791 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.071750    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" podUID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" containerName="main" containerID="cri-o://a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229" gracePeriod=45
Sep 11 21:42:50.361886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.361764    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:42:50.391056 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.391026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:42:50.396995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.396973    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:42:50.401162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.401140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:42:50.476184 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.476353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.476353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476230    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4khk\" (UniqueName: \"kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.476353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476267    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.476353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hqxk\" (UniqueName: \"kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.476353 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476364    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476392    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6lqs\" (UniqueName: \"kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home" (OuterVolumeSpecName: "home") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.476645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home" (OuterVolumeSpecName: "home") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.476991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.476991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.476991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.476981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache" (OuterVolumeSpecName: "model-cache") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache\") pod \"edd3c8af-36ff-40fe-be3c-13aa48298e39\" (UID: \"edd3c8af-36ff-40fe-be3c-13aa48298e39\") "
Sep 11 21:42:50.477149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.477535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.477535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.477535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs\") pod \"ce9ba619-f092-4c3a-8348-51e69a961b59\" (UID: \"ce9ba619-f092-4c3a-8348-51e69a961b59\") "
Sep 11 21:42:50.477535 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477528    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.477767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home\") pod \"85549863-90e8-42b0-aaf6-e45573a66410\" (UID: \"85549863-90e8-42b0-aaf6-e45573a66410\") "
Sep 11 21:42:50.477767 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx746\" (UniqueName: \"kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746\") pod \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\" (UID: \"f0214f63-8183-47d1-ab5c-c0ddf8f081b7\") "
Sep 11 21:42:50.478639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.478163    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.478639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.478187    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.478639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.478226    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.478639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.478241    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.478639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.478253    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.479619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home" (OuterVolumeSpecName: "home") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.479619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.479619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.479619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.477650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache" (OuterVolumeSpecName: "model-cache") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.479619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.479547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm" (OuterVolumeSpecName: "dshm") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.480278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.479825    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.481807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.481548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm" (OuterVolumeSpecName: "dshm") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.481807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.481621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:50.481970 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.481808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home" (OuterVolumeSpecName: "home") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.482117 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.482083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.482570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.482534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:50.482671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.482574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:50.482881 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.482831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk" (OuterVolumeSpecName: "kube-api-access-f4khk") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "kube-api-access-f4khk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:50.482957 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.482916    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:42:50.483162 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.483135    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk" (OuterVolumeSpecName: "kube-api-access-7hqxk") pod "edd3c8af-36ff-40fe-be3c-13aa48298e39" (UID: "edd3c8af-36ff-40fe-be3c-13aa48298e39"). InnerVolumeSpecName "kube-api-access-7hqxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:50.483301 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.483280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm" (OuterVolumeSpecName: "dshm") pod "ce9ba619-f092-4c3a-8348-51e69a961b59" (UID: "ce9ba619-f092-4c3a-8348-51e69a961b59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.483402 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.483376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs" (OuterVolumeSpecName: "kube-api-access-d6lqs") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "kube-api-access-d6lqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:50.483702 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.483681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm" (OuterVolumeSpecName: "dshm") pod "85549863-90e8-42b0-aaf6-e45573a66410" (UID: "85549863-90e8-42b0-aaf6-e45573a66410"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:50.484478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.484457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746" (OuterVolumeSpecName: "kube-api-access-gx746") pod "f0214f63-8183-47d1-ab5c-c0ddf8f081b7" (UID: "f0214f63-8183-47d1-ab5c-c0ddf8f081b7"). InnerVolumeSpecName "kube-api-access-gx746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:50.579701 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579671    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edd3c8af-36ff-40fe-be3c-13aa48298e39-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579701 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579696    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579709    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579719    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579729    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85549863-90e8-42b0-aaf6-e45573a66410-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579738    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579748    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edd3c8af-36ff-40fe-be3c-13aa48298e39-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579759    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579768    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579779    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579791    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9ba619-f092-4c3a-8348-51e69a961b59-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579801    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579811    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85549863-90e8-42b0-aaf6-e45573a66410-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579823    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx746\" (UniqueName: \"kubernetes.io/projected/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-kube-api-access-gx746\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579838    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4khk\" (UniqueName: \"kubernetes.io/projected/ce9ba619-f092-4c3a-8348-51e69a961b59-kube-api-access-f4khk\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579851    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9ba619-f092-4c3a-8348-51e69a961b59-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqxk\" (UniqueName: \"kubernetes.io/projected/edd3c8af-36ff-40fe-be3c-13aa48298e39-kube-api-access-7hqxk\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579877    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0214f63-8183-47d1-ab5c-c0ddf8f081b7-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:50.579888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:50.579888    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lqs\" (UniqueName: \"kubernetes.io/projected/85549863-90e8-42b0-aaf6-e45573a66410-kube-api-access-d6lqs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:51.107577 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.107548    2580 generic.go:356] "Generic (PLEG): container finished" podID="85549863-90e8-42b0-aaf6-e45573a66410" containerID="94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce" exitCode=0
Sep 11 21:42:51.107964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.107596    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"
Sep 11 21:42:51.107964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.107636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" event={"ID":"85549863-90e8-42b0-aaf6-e45573a66410","Type":"ContainerDied","Data":"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"}
Sep 11 21:42:51.107964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.107687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn" event={"ID":"85549863-90e8-42b0-aaf6-e45573a66410","Type":"ContainerDied","Data":"16425b32d7cb7aab1ba1b11f230f361b6ffb8b5fef9c0beded6e3a69870f9619"}
Sep 11 21:42:51.107964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.107711    2580 scope.go:117] "RemoveContainer" containerID="94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"
Sep 11 21:42:51.109100 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.109019    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" containerID="a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229" exitCode=0
Sep 11 21:42:51.109170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.109112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"
Sep 11 21:42:51.109226 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.109109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" event={"ID":"f0214f63-8183-47d1-ab5c-c0ddf8f081b7","Type":"ContainerDied","Data":"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"}
Sep 11 21:42:51.109282 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.109226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25" event={"ID":"f0214f63-8183-47d1-ab5c-c0ddf8f081b7","Type":"ContainerDied","Data":"8275e5eb9a8d9f30bd39d2c33467e28bf7d05fc4134658d940b362b288231416"}
Sep 11 21:42:51.110587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.110565    2580 generic.go:356] "Generic (PLEG): container finished" podID="edd3c8af-36ff-40fe-be3c-13aa48298e39" containerID="0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40" exitCode=0
Sep 11 21:42:51.110704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.110620    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"
Sep 11 21:42:51.110704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.110656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" event={"ID":"edd3c8af-36ff-40fe-be3c-13aa48298e39","Type":"ContainerDied","Data":"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"}
Sep 11 21:42:51.110704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.110683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9" event={"ID":"edd3c8af-36ff-40fe-be3c-13aa48298e39","Type":"ContainerDied","Data":"859cccb9fd6fe88fef43db06f6c59d1f45a9f33daca13cf79505cf6a63003bc3"}
Sep 11 21:42:51.112306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.112212    2580 generic.go:356] "Generic (PLEG): container finished" podID="ce9ba619-f092-4c3a-8348-51e69a961b59" containerID="b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817" exitCode=0
Sep 11 21:42:51.112306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.112257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" event={"ID":"ce9ba619-f092-4c3a-8348-51e69a961b59","Type":"ContainerDied","Data":"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"}
Sep 11 21:42:51.112306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.112277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449" event={"ID":"ce9ba619-f092-4c3a-8348-51e69a961b59","Type":"ContainerDied","Data":"c9ea3779c252116072f96c659e327ca03f02840f8085408043deb359f8ceb991"}
Sep 11 21:42:51.112306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.112258    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"
Sep 11 21:42:51.118338 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.118323    2580 scope.go:117] "RemoveContainer" containerID="94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"
Sep 11 21:42:51.118666 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:51.118646    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce\": container with ID starting with 94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce not found: ID does not exist" containerID="94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"
Sep 11 21:42:51.118745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.118672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce"} err="failed to get container status \"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce\": rpc error: code = NotFound desc = could not find container \"94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce\": container with ID starting with 94db2ec10ca66f522c8d3fc14cead861d077d143c05416e4d14c8d436d3b2dce not found: ID does not exist"
Sep 11 21:42:51.118745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.118687    2580 scope.go:117] "RemoveContainer" containerID="a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"
Sep 11 21:42:51.127458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.127442    2580 scope.go:117] "RemoveContainer" containerID="a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"
Sep 11 21:42:51.127798 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:51.127778    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229\": container with ID starting with a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229 not found: ID does not exist" containerID="a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"
Sep 11 21:42:51.127877 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.127804    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229"} err="failed to get container status \"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229\": rpc error: code = NotFound desc = could not find container \"a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229\": container with ID starting with a4c495a03c9bd14928ef63094f67073117180e6d2e5b98ec34ebb7c058dc2229 not found: ID does not exist"
Sep 11 21:42:51.127877 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.127818    2580 scope.go:117] "RemoveContainer" containerID="0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"
Sep 11 21:42:51.136273 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.136208    2580 scope.go:117] "RemoveContainer" containerID="0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"
Sep 11 21:42:51.136591 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.136570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:42:51.136735 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:51.136583    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40\": container with ID starting with 0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40 not found: ID does not exist" containerID="0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"
Sep 11 21:42:51.136841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.136745    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40"} err="failed to get container status \"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40\": rpc error: code = NotFound desc = could not find container \"0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40\": container with ID starting with 0d280553856df39ad0f59156ff84cc0ebecf8e307d7b592d5d49fa52d7abbb40 not found: ID does not exist"
Sep 11 21:42:51.136841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.136767    2580 scope.go:117] "RemoveContainer" containerID="b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"
Sep 11 21:42:51.138934 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.138915    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-w4-83ba87-kserve-7c8c47b9bb-sg2hn"]
Sep 11 21:42:51.145471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.145456    2580 scope.go:117] "RemoveContainer" containerID="b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"
Sep 11 21:42:51.145752 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:51.145736    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817\": container with ID starting with b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817 not found: ID does not exist" containerID="b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"
Sep 11 21:42:51.145805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.145756    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817"} err="failed to get container status \"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817\": rpc error: code = NotFound desc = could not find container \"b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817\": container with ID starting with b05c86493ffa0c8a64aa49d3be85de7c4958cd36643062f9433a57f97e8be817 not found: ID does not exist"
Sep 11 21:42:51.149633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.149615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:42:51.153996 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.153977    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/distinct-2-w4-83ba87-kserve-7d464d4b7c-92449"]
Sep 11 21:42:51.176522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.171431    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:42:51.176522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.176287    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/simulator-w4-83ba87-kserve-6b646c64c4-rlx25"]
Sep 11 21:42:51.188711 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.188689    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:42:51.192358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:51.192340    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-83ba87/premium-w4-83ba87-kserve-66c596bf4c-h74t9"]
Sep 11 21:42:52.383600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:52.383569    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85549863-90e8-42b0-aaf6-e45573a66410" path="/var/lib/kubelet/pods/85549863-90e8-42b0-aaf6-e45573a66410/volumes"
Sep 11 21:42:52.383954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:52.383946    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9ba619-f092-4c3a-8348-51e69a961b59" path="/var/lib/kubelet/pods/ce9ba619-f092-4c3a-8348-51e69a961b59/volumes"
Sep 11 21:42:52.384265 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:52.384253    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edd3c8af-36ff-40fe-be3c-13aa48298e39" path="/var/lib/kubelet/pods/edd3c8af-36ff-40fe-be3c-13aa48298e39/volumes"
Sep 11 21:42:52.384881 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:52.384867    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" path="/var/lib/kubelet/pods/f0214f63-8183-47d1-ab5c-c0ddf8f081b7/volumes"
Sep 11 21:42:53.378145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:53.378115    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:42:53.378347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:53.378320    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy" containerID="cri-o://b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f" gracePeriod=30
Sep 11 21:42:55.593080 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:55.593049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:42:57.672075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:57.672043    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:42:57.672428 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:57.672302    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" containerID="cri-o://c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af" gracePeriod=30
Sep 11 21:42:58.424540 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7810-5802dc"]
Sep 11 21:42:58.424870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424857    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2992f556-45ab-4435-8b97-940dc68e803e" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2992f556-45ab-4435-8b97-940dc68e803e" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424880    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9ba619-f092-4c3a-8348-51e69a961b59" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424885    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9ba619-f092-4c3a-8348-51e69a961b59" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424895    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85549863-90e8-42b0-aaf6-e45573a66410" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424900    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85549863-90e8-42b0-aaf6-e45573a66410" containerName="main"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424905    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.424914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424910    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424917    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424922    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424929    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424934    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424940    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97477337-3245-4fed-9eea-0a5ae1103e74" containerName="maas-api"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424946    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="97477337-3245-4fed-9eea-0a5ae1103e74" containerName="maas-api"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424954    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd3c8af-36ff-40fe-be3c-13aa48298e39" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd3c8af-36ff-40fe-be3c-13aa48298e39" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424966    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424971    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424976    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424980    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424992    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.424997    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425005    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed9750a-a332-42d8-99f9-5195a692470a" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed9750a-a332-42d8-99f9-5195a692470a" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425015    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425019    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd3c8af-36ff-40fe-be3c-13aa48298e39" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425087    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9ba619-f092-4c3a-8348-51e69a961b59" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425092    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="772ef7ba-034e-446d-ac99-b0f4f5190d7e" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425098    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425103    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425107    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c97730-7b30-4ec2-8822-f521196470fb" containerName="istio-proxy"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425113    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed9750a-a332-42d8-99f9-5195a692470a" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425118    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0214f63-8183-47d1-ab5c-c0ddf8f081b7" containerName="main"
Sep 11 21:42:58.425111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425123    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2992f556-45ab-4435-8b97-940dc68e803e" containerName="main"
Sep 11 21:42:58.425884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425127    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="97477337-3245-4fed-9eea-0a5ae1103e74" containerName="maas-api"
Sep 11 21:42:58.425884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425132    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85549863-90e8-42b0-aaf6-e45573a66410" containerName="main"
Sep 11 21:42:58.425884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.425136    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f227fd7c-24a0-4e71-93b7-4e4bc0f3b646" containerName="main"
Sep 11 21:42:58.428085 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.428066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:42:58.430531 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.430510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ns82b\""
Sep 11 21:42:58.435257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.435238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7810-5802dc"]
Sep 11 21:42:58.548274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.548238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szm6x\" (UniqueName: \"kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x\") pod \"test-curl-7810-5802dc\" (UID: \"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b\") " pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:42:58.628423 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.628394    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:42:58.649222 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.649169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szm6x\" (UniqueName: \"kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x\") pod \"test-curl-7810-5802dc\" (UID: \"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b\") " pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:42:58.658841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.658814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szm6x\" (UniqueName: \"kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x\") pod \"test-curl-7810-5802dc\" (UID: \"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b\") " pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:42:58.749580 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749656    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brbc\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert\") pod \"0a75ab54-e867-4ade-8663-a176c9b6ea25\" (UID: \"0a75ab54-e867-4ade-8663-a176c9b6ea25\") "
Sep 11 21:42:58.749916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.749964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.750104    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.750126    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.750140    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.750142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:42:58.750349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.750146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:58.751886 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.751862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:58.752038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.752020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:42:58.752439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.752412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:42:58.752439 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.752413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc" (OuterVolumeSpecName: "kube-api-access-9brbc") pod "0a75ab54-e867-4ade-8663-a176c9b6ea25" (UID: "0a75ab54-e867-4ade-8663-a176c9b6ea25"). InnerVolumeSpecName "kube-api-access-9brbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:42:58.766705 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.766688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:42:58.851362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851336    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.851362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851360    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brbc\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-kube-api-access-9brbc\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.851362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851369    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.851582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851377    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a75ab54-e867-4ade-8663-a176c9b6ea25-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.851582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851384    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:58.851582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:58.851392    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a75ab54-e867-4ade-8663-a176c9b6ea25-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:42:59.034456 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.034430    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7810-5802dc"]
Sep 11 21:42:59.036245 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:42:59.036215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6edb164e_f2e0_40eb_b5c0_3ecae6ee6d2b.slice/crio-89ea40c68a8ca82aeff86e56ba39d1ada4d361fdca6a0f42731e88398cdd476a WatchSource:0}: Error finding container 89ea40c68a8ca82aeff86e56ba39d1ada4d361fdca6a0f42731e88398cdd476a: Status 404 returned error can't find the container with id 89ea40c68a8ca82aeff86e56ba39d1ada4d361fdca6a0f42731e88398cdd476a
Sep 11 21:42:59.157417 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.157392    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerID="b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f" exitCode=0
Sep 11 21:42:59.157549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.157437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"
Sep 11 21:42:59.157549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.157465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerDied","Data":"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"}
Sep 11 21:42:59.157549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.157522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx" event={"ID":"0a75ab54-e867-4ade-8663-a176c9b6ea25","Type":"ContainerDied","Data":"ae20b0cb1ee5c4f168d03913440a6a68feaccd388d4947b590adc8966667d28f"}
Sep 11 21:42:59.157687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.157549    2580 scope.go:117] "RemoveContainer" containerID="b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"
Sep 11 21:42:59.158802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.158773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7810-5802dc" event={"ID":"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b","Type":"ContainerStarted","Data":"89ea40c68a8ca82aeff86e56ba39d1ada4d361fdca6a0f42731e88398cdd476a"}
Sep 11 21:42:59.166249 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.166234    2580 scope.go:117] "RemoveContainer" containerID="2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"
Sep 11 21:42:59.174911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.174732    2580 scope.go:117] "RemoveContainer" containerID="b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"
Sep 11 21:42:59.175998 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:59.175965    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f\": container with ID starting with b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f not found: ID does not exist" containerID="b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"
Sep 11 21:42:59.175998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.175992    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f"} err="failed to get container status \"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f\": rpc error: code = NotFound desc = could not find container \"b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f\": container with ID starting with b8113bc41cc2a03da64905f33ce696038a55b494fcf29560b97d38b4b13dc89f not found: ID does not exist"
Sep 11 21:42:59.176132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.176007    2580 scope.go:117] "RemoveContainer" containerID="2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"
Sep 11 21:42:59.176256 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:42:59.176240    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e\": container with ID starting with 2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e not found: ID does not exist" containerID="2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"
Sep 11 21:42:59.176300 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.176261    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e"} err="failed to get container status \"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e\": rpc error: code = NotFound desc = could not find container \"2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e\": container with ID starting with 2488b9f22eb35331724e61813acc790da063ad79f12d2dd46f34f321cc99254e not found: ID does not exist"
Sep 11 21:42:59.178420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.178402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:42:59.184483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:42:59.184458    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-83ba87-openshift-default-8f9947d4b-bnknx"]
Sep 11 21:43:00.183418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:00.183383    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:43:00.183890 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:00.183450    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:43:00.384416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:00.384384    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" path="/var/lib/kubelet/pods/0a75ab54-e867-4ade-8663-a176c9b6ea25/volumes"
Sep 11 21:43:01.172084 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:01.172058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7810-5802dc" event={"ID":"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b","Type":"ContainerStarted","Data":"269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6"}
Sep 11 21:43:01.185947 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:01.185880    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7810-5802dc" podStartSLOduration=1.160321846 podStartE2EDuration="3.185858988s" totalImagesPullingTime="2.025537142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:01.185272276 +0000 UTC m=+1529.307604196" watchObservedRunningTime="2026-09-11 21:43:01.185858988 +0000 UTC m=+1529.308190910"
Sep 11 21:43:01.800689 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:01.799337    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7810-5802dc"]
Sep 11 21:43:01.805415 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:01.805388    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7810-5802dc"]
Sep 11 21:43:02.177234 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:43:02.177159    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7810-5802dc" container="test-curl-7810-5802dc"
Sep 11 21:43:02.918579 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:02.918559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:43:03.089182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089183    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089258    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089313    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089373 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089402    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlhbf\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf\") pod \"a830f849-1a60-440f-a19e-9b4a4a8db382\" (UID: \"a830f849-1a60-440f-a19e-9b4a4a8db382\") "
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data" (OuterVolumeSpecName: "istio-data") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:43:03.089692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:43:03.089952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089746    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.089952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089763    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a830f849-1a60-440f-a19e-9b4a4a8db382-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.089952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089776    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.089952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089788    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.089952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.089801    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.091543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.091516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:43:03.091661 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.091559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token" (OuterVolumeSpecName: "istio-token") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:43:03.092078 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.092051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf" (OuterVolumeSpecName: "kube-api-access-dlhbf") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "kube-api-access-dlhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:43:03.092078 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.092054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a830f849-1a60-440f-a19e-9b4a4a8db382" (UID: "a830f849-1a60-440f-a19e-9b4a4a8db382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:43:03.183454 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183422    2580 generic.go:356] "Generic (PLEG): container finished" podID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerID="c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af" exitCode=0
Sep 11 21:43:03.183587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"
Sep 11 21:43:03.183587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerDied","Data":"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"}
Sep 11 21:43:03.183587 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183577    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6" event={"ID":"a830f849-1a60-440f-a19e-9b4a4a8db382","Type":"ContainerDied","Data":"0622ca5da874e86a4952d2a389bc25308edef6e1608a263c46469f6a6d154566"}
Sep 11 21:43:03.183693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183603    2580 scope.go:117] "RemoveContainer" containerID="c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"
Sep 11 21:43:03.183910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.183875    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7810-5802dc" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" containerName="test-curl-7810-5802dc" containerID="cri-o://269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6" gracePeriod=2
Sep 11 21:43:03.190463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.190444    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhbf\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-kube-api-access-dlhbf\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.190463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.190464    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.190644 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.190473    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.190644 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.190481    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a830f849-1a60-440f-a19e-9b4a4a8db382-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:03.193616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.193590    2580 scope.go:117] "RemoveContainer" containerID="4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"
Sep 11 21:43:03.201396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.201376    2580 scope.go:117] "RemoveContainer" containerID="c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"
Sep 11 21:43:03.201706 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:43:03.201687    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af\": container with ID starting with c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af not found: ID does not exist" containerID="c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"
Sep 11 21:43:03.201773 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.201714    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af"} err="failed to get container status \"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af\": rpc error: code = NotFound desc = could not find container \"c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af\": container with ID starting with c8747723541cb64a3691120f55bd77c3e19305016f77cd34af45b0c844c323af not found: ID does not exist"
Sep 11 21:43:03.201773 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.201728    2580 scope.go:117] "RemoveContainer" containerID="4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"
Sep 11 21:43:03.202012 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:43:03.201991    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549\": container with ID starting with 4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549 not found: ID does not exist" containerID="4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"
Sep 11 21:43:03.202059 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.202020    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549"} err="failed to get container status \"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549\": rpc error: code = NotFound desc = could not find container \"4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549\": container with ID starting with 4f391f877cd0acbfcbf2b164fe413ccaf6e8882e9b471c8d00bd9a73b361f549 not found: ID does not exist"
Sep 11 21:43:03.206693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.206666    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:43:03.210812 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:03.210793    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-408219-openshift-default-6f9d8564dd-sv4h6"]
Sep 11 21:43:04.383361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:04.383330    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" path="/var/lib/kubelet/pods/a830f849-1a60-440f-a19e-9b4a4a8db382/volumes"
Sep 11 21:43:05.350987 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:05.350963    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:43:05.510261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:05.510237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szm6x\" (UniqueName: \"kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x\") pod \"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b\" (UID: \"6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b\") "
Sep 11 21:43:05.512521 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:05.512486    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x" (OuterVolumeSpecName: "kube-api-access-szm6x") pod "6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" (UID: "6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b"). InnerVolumeSpecName "kube-api-access-szm6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:43:05.610962 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:05.610923    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szm6x\" (UniqueName: \"kubernetes.io/projected/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b-kube-api-access-szm6x\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:43:06.199619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.199589    2580 generic.go:356] "Generic (PLEG): container finished" podID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" containerID="269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6" exitCode=137
Sep 11 21:43:06.199802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.199627    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:43:06.199802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.199642    2580 scope.go:117] "RemoveContainer" containerID="269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6"
Sep 11 21:43:06.208186 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.208172    2580 scope.go:117] "RemoveContainer" containerID="269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6"
Sep 11 21:43:06.208457 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:43:06.208434    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6\": container with ID starting with 269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6 not found: ID does not exist" containerID="269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6"
Sep 11 21:43:06.208573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.208462    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6"} err="failed to get container status \"269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6\": rpc error: code = NotFound desc = could not find container \"269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6\": container with ID starting with 269ca30fca79e973338e61c5dd7e6a320af8ab20c7584adb154dc46e59a0a3e6 not found: ID does not exist"
Sep 11 21:43:06.215084 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:43:06.215063    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7810-5802dc\" is forbidden: User \"system:node:ip-10-0-135-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-133.ec2.internal' and this object" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" pod="opendatahub/test-curl-7810-5802dc"
Sep 11 21:43:06.383111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:06.383077    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" path="/var/lib/kubelet/pods/6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b/volumes"
Sep 11 21:43:14.083646 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:14.083615    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:43:38.693272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693236    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693685    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693710    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693720    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693740    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693753    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693761    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693770    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" containerName="test-curl-7810-5802dc"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693779    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" containerName="test-curl-7810-5802dc"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693863    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693876    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edb164e-f2e0-40eb-b5c0-3ecae6ee6d2b" containerName="test-curl-7810-5802dc"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693887    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="740c22fe-f3cf-4495-9402-153516b09a19" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693896    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.694008 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.693905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a75ab54-e867-4ade-8663-a176c9b6ea25" containerName="istio-proxy"
Sep 11 21:43:38.699796 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.699776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.703163 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.703145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c97420\"/\"sub-test-model-c97420-kserve-self-signed-certs\""
Sep 11 21:43:38.703267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.703166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c97420\"/\"openshift-service-ca.crt\""
Sep 11 21:43:38.703267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.703146    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c97420\"/\"default-dockercfg-88njg\""
Sep 11 21:43:38.703267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.703244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c97420\"/\"kube-root-ca.crt\""
Sep 11 21:43:38.706190 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.706169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:43:38.765735 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.765846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.765846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.765926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.765926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgpn\" (UniqueName: \"kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.765926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.765908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866270 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgpn\" (UniqueName: \"kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866721 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.866769 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.866756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.868753 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.868731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.868902 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.868887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:38.874458 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:38.874436    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgpn\" (UniqueName: \"kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn\") pod \"sub-test-model-c97420-kserve-855f5dc7b9-qb6bw\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") " pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:39.009482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:39.009458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:39.283770 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:39.283698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:43:39.287072 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:43:39.287045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e7b9016_cb2c_420f_ac6e_bf687042828f.slice/crio-5b23c2bf92bd633986fd40fed4efe9fb9e7325eee6b19e5b60d9b14721d94eaa WatchSource:0}: Error finding container 5b23c2bf92bd633986fd40fed4efe9fb9e7325eee6b19e5b60d9b14721d94eaa: Status 404 returned error can't find the container with id 5b23c2bf92bd633986fd40fed4efe9fb9e7325eee6b19e5b60d9b14721d94eaa
Sep 11 21:43:39.368352 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:39.368326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" event={"ID":"1e7b9016-cb2c-420f-ac6e-bf687042828f","Type":"ContainerStarted","Data":"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"}
Sep 11 21:43:39.368469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:39.368363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" event={"ID":"1e7b9016-cb2c-420f-ac6e-bf687042828f","Type":"ContainerStarted","Data":"5b23c2bf92bd633986fd40fed4efe9fb9e7325eee6b19e5b60d9b14721d94eaa"}
Sep 11 21:43:39.387460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:39.387420    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" podStartSLOduration=1.387399357 podStartE2EDuration="1.387399357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:39.386130519 +0000 UTC m=+1567.508462439" watchObservedRunningTime="2026-09-11 21:43:39.387399357 +0000 UTC m=+1567.509731277"
Sep 11 21:43:41.599892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:41.599856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:43:49.010388 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:49.010351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:49.010839 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:49.010401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:49.022619 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:49.022596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:49.426777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:49.426701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:43:52.187639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.187607    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:43:52.188020 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.188010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a830f849-1a60-440f-a19e-9b4a4a8db382" containerName="istio-proxy"
Sep 11 21:43:52.191295 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.191276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.194829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.194787    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"openshift-service-ca.crt\""
Sep 11 21:43:52.194829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.194801    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"sub-test-model-e0c211-kserve-self-signed-certs\""
Sep 11 21:43:52.194829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.194824    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"kube-root-ca.crt\""
Sep 11 21:43:52.195105 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.194787    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"default-dockercfg-mqtsg\""
Sep 11 21:43:52.201011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.200988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:43:52.268254 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.268392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.268392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.268392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.268392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hp9\" (UniqueName: \"kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.268617 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.268450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.369748 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.369885 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.369885 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.369885 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.370042 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.370042 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.369991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hp9\" (UniqueName: \"kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.370219 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.370186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.370358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.370280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.370358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.370309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.372149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.372127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.372407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.372389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.378785 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.378749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hp9\" (UniqueName: \"kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9\") pod \"sub-test-model-e0c211-kserve-74787d8987-sz7vn\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.502605 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.502581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:43:52.774389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:52.774322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:43:52.776463 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:43:52.776434    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab987cdd_2fb4_447b_b11e_98775283e0c8.slice/crio-bb4b33e7f4898e36b55b052495a8c7f68d9ee46545df8a136a375097169e544b WatchSource:0}: Error finding container bb4b33e7f4898e36b55b052495a8c7f68d9ee46545df8a136a375097169e544b: Status 404 returned error can't find the container with id bb4b33e7f4898e36b55b052495a8c7f68d9ee46545df8a136a375097169e544b
Sep 11 21:43:53.438039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:53.438009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" event={"ID":"ab987cdd-2fb4-447b-b11e-98775283e0c8","Type":"ContainerStarted","Data":"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"}
Sep 11 21:43:53.438401 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:53.438048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" event={"ID":"ab987cdd-2fb4-447b-b11e-98775283e0c8","Type":"ContainerStarted","Data":"bb4b33e7f4898e36b55b052495a8c7f68d9ee46545df8a136a375097169e544b"}
Sep 11 21:43:53.458557 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:53.458489    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" podStartSLOduration=1.458473957 podStartE2EDuration="1.458473957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:53.456597079 +0000 UTC m=+1581.578928999" watchObservedRunningTime="2026-09-11 21:43:53.458473957 +0000 UTC m=+1581.580805876"
Sep 11 21:43:55.647757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:43:55.647722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:44:02.503582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:02.503542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:02.503959 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:02.503593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:02.516154 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:02.516133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:03.491122 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:03.491093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:11.497805 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:11.497770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:44:23.989829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:23.989752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:44:32.596973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:32.596938    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:44:33.582797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:33.582758    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:44:36.196361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.196332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:44:36.746991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.746956    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:44:36.750342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.750324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.754725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.754708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c97420\"/\"rate-test-model-c97420-kserve-self-signed-certs\""
Sep 11 21:44:36.821564 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.821663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rm5\" (UniqueName: \"kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.821663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.821774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.821774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.821874 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.821783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.840543 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.840514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:44:36.922471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rm5\" (UniqueName: \"kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922905 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922972 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.922972 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.922961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.924936 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.924916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.925111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.925092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:36.937278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:36.937253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rm5\" (UniqueName: \"kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5\") pod \"rate-test-model-c97420-kserve-6d6cf96567-9fklj\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") " pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:37.059475 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:37.059402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:37.333065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:37.333038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:44:37.335920 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:44:37.335893    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode933b1e3_92e2_4899_b4a4_0b535a823a3a.slice/crio-55513a01e9619286c977fcd2aa0009866824b43d7e0ecf60e927c39af330b17b WatchSource:0}: Error finding container 55513a01e9619286c977fcd2aa0009866824b43d7e0ecf60e927c39af330b17b: Status 404 returned error can't find the container with id 55513a01e9619286c977fcd2aa0009866824b43d7e0ecf60e927c39af330b17b
Sep 11 21:44:37.651611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:37.651532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" event={"ID":"e933b1e3-92e2-4899-b4a4-0b535a823a3a","Type":"ContainerStarted","Data":"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"}
Sep 11 21:44:37.651611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:37.651578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" event={"ID":"e933b1e3-92e2-4899-b4a4-0b535a823a3a","Type":"ContainerStarted","Data":"55513a01e9619286c977fcd2aa0009866824b43d7e0ecf60e927c39af330b17b"}
Sep 11 21:44:37.669819 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:37.669773    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" podStartSLOduration=1.669757505 podStartE2EDuration="1.669757505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:37.669133155 +0000 UTC m=+1625.791465077" watchObservedRunningTime="2026-09-11 21:44:37.669757505 +0000 UTC m=+1625.792089424"
Sep 11 21:44:44.892667 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:44.892635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:44:47.059816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.059787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:47.060252 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.059868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:47.072261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.072235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:47.616418 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.616381    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" podUID="1e7b9016-cb2c-420f-ac6e-bf687042828f" containerName="main" containerID="cri-o://718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c" gracePeriod=45
Sep 11 21:44:47.706943 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.706914    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:44:47.861984 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.861960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:44:47.907998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.907927    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.907998 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.907990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.908199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908024    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.908199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.908199 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.908368 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbgpn\" (UniqueName: \"kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn\") pod \"1e7b9016-cb2c-420f-ac6e-bf687042828f\" (UID: \"1e7b9016-cb2c-420f-ac6e-bf687042828f\") "
Sep 11 21:44:47.908368 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:47.908368 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home" (OuterVolumeSpecName: "home") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:47.908537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:47.908537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908487    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:47.908537 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.908525    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:47.910560 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.910538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn" (OuterVolumeSpecName: "kube-api-access-xbgpn") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "kube-api-access-xbgpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:47.910650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.910610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm" (OuterVolumeSpecName: "dshm") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:47.910703 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:47.910675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e7b9016-cb2c-420f-ac6e-bf687042828f" (UID: "1e7b9016-cb2c-420f-ac6e-bf687042828f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:44:48.009546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.009521    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7b9016-cb2c-420f-ac6e-bf687042828f-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.009546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.009542    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.009546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.009549    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7b9016-cb2c-420f-ac6e-bf687042828f-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.009712 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.009557    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbgpn\" (UniqueName: \"kubernetes.io/projected/1e7b9016-cb2c-420f-ac6e-bf687042828f-kube-api-access-xbgpn\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.601243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.601211    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" podUID="ab987cdd-2fb4-447b-b11e-98775283e0c8" containerName="main" containerID="cri-o://8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f" gracePeriod=45
Sep 11 21:44:48.701783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.701758    2580 generic.go:356] "Generic (PLEG): container finished" podID="1e7b9016-cb2c-420f-ac6e-bf687042828f" containerID="718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c" exitCode=0
Sep 11 21:44:48.701901 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.701810    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"
Sep 11 21:44:48.701901 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.701845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" event={"ID":"1e7b9016-cb2c-420f-ac6e-bf687042828f","Type":"ContainerDied","Data":"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"}
Sep 11 21:44:48.701901 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.701886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw" event={"ID":"1e7b9016-cb2c-420f-ac6e-bf687042828f","Type":"ContainerDied","Data":"5b23c2bf92bd633986fd40fed4efe9fb9e7325eee6b19e5b60d9b14721d94eaa"}
Sep 11 21:44:48.702018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.701908    2580 scope.go:117] "RemoveContainer" containerID="718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"
Sep 11 21:44:48.711895 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.711869    2580 scope.go:117] "RemoveContainer" containerID="718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"
Sep 11 21:44:48.712157 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:44:48.712141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c\": container with ID starting with 718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c not found: ID does not exist" containerID="718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"
Sep 11 21:44:48.712215 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.712162    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c"} err="failed to get container status \"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c\": rpc error: code = NotFound desc = could not find container \"718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c\": container with ID starting with 718132c0640f2402bb8255e157bf2466c344e0e79ff5330f07a9eb42a9f1a62c not found: ID does not exist"
Sep 11 21:44:48.721979 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.721956    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:44:48.726140 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.726121    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/sub-test-model-c97420-kserve-855f5dc7b9-qb6bw"]
Sep 11 21:44:48.846407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.846378    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:48.916637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.916637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4hp9\" (UniqueName: \"kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.916822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.916822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.916822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.916822 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs\") pod \"ab987cdd-2fb4-447b-b11e-98775283e0c8\" (UID: \"ab987cdd-2fb4-447b-b11e-98775283e0c8\") "
Sep 11 21:44:48.917013 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home" (OuterVolumeSpecName: "home") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:48.917013 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:48.917013 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.916978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:48.917126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.917080    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.917126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.917096    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.917126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.917105    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:48.918957 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.918937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:44:48.919031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.918972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm" (OuterVolumeSpecName: "dshm") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:44:48.919093 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:48.919078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9" (OuterVolumeSpecName: "kube-api-access-z4hp9") pod "ab987cdd-2fb4-447b-b11e-98775283e0c8" (UID: "ab987cdd-2fb4-447b-b11e-98775283e0c8"). InnerVolumeSpecName "kube-api-access-z4hp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:49.017631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.017608    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hp9\" (UniqueName: \"kubernetes.io/projected/ab987cdd-2fb4-447b-b11e-98775283e0c8-kube-api-access-z4hp9\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:49.017631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.017628    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab987cdd-2fb4-447b-b11e-98775283e0c8-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:49.017766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.017638    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab987cdd-2fb4-447b-b11e-98775283e0c8-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:44:49.707956 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.707920    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab987cdd-2fb4-447b-b11e-98775283e0c8" containerID="8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f" exitCode=0
Sep 11 21:44:49.708387 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.707963    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"
Sep 11 21:44:49.708387 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.708004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" event={"ID":"ab987cdd-2fb4-447b-b11e-98775283e0c8","Type":"ContainerDied","Data":"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"}
Sep 11 21:44:49.708387 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.708042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn" event={"ID":"ab987cdd-2fb4-447b-b11e-98775283e0c8","Type":"ContainerDied","Data":"bb4b33e7f4898e36b55b052495a8c7f68d9ee46545df8a136a375097169e544b"}
Sep 11 21:44:49.708387 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.708066    2580 scope.go:117] "RemoveContainer" containerID="8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"
Sep 11 21:44:49.716774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.716756    2580 scope.go:117] "RemoveContainer" containerID="8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"
Sep 11 21:44:49.717019 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:44:49.716999    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f\": container with ID starting with 8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f not found: ID does not exist" containerID="8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"
Sep 11 21:44:49.717086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.717029    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f"} err="failed to get container status \"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f\": rpc error: code = NotFound desc = could not find container \"8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f\": container with ID starting with 8408ee0665d120d606fbbeb91188d31d050545bf3d54c987ef81e39011d2f82f not found: ID does not exist"
Sep 11 21:44:49.728390 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.728361    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:44:49.731872 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:49.731854    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/sub-test-model-e0c211-kserve-74787d8987-sz7vn"]
Sep 11 21:44:50.383050 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:50.383006    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e7b9016-cb2c-420f-ac6e-bf687042828f" path="/var/lib/kubelet/pods/1e7b9016-cb2c-420f-ac6e-bf687042828f/volumes"
Sep 11 21:44:50.383420 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:50.383406    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab987cdd-2fb4-447b-b11e-98775283e0c8" path="/var/lib/kubelet/pods/ab987cdd-2fb4-447b-b11e-98775283e0c8/volumes"
Sep 11 21:44:58.881994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.881964    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882277    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e7b9016-cb2c-420f-ac6e-bf687042828f" containerName="main"
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882287    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e7b9016-cb2c-420f-ac6e-bf687042828f" containerName="main"
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882298    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab987cdd-2fb4-447b-b11e-98775283e0c8" containerName="main"
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882303    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab987cdd-2fb4-447b-b11e-98775283e0c8" containerName="main"
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882350    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab987cdd-2fb4-447b-b11e-98775283e0c8" containerName="main"
Sep 11 21:44:58.882363 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.882358    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e7b9016-cb2c-420f-ac6e-bf687042828f" containerName="main"
Sep 11 21:44:58.886791 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.886769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.889335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.889313    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"openshift-service-ca.crt\""
Sep 11 21:44:58.889440 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.889313    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"default-dockercfg-mqtsg\""
Sep 11 21:44:58.890299 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.890281    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"kube-root-ca.crt\""
Sep 11 21:44:58.890371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.890317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e0c211\"/\"rate-test-model-e0c211-kserve-self-signed-certs\""
Sep 11 21:44:58.897206 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.897186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:44:58.987247 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.987360 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.987360 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.987476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flm2\" (UniqueName: \"kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.987476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:58.987476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:58.987443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088582 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088697 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088801 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flm2\" (UniqueName: \"kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.088801 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.088764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.089020 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.089000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.089068 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.089046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.089110 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.089066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.090991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.090968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.091365 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.091348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.098592 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.098573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flm2\" (UniqueName: \"kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2\") pod \"rate-test-model-e0c211-kserve-f6b4675d7-rh6xl\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") " pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.196758 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.196697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:44:59.470821 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.470730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:44:59.478005 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:44:59.477964    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48558443_fa02_4878_b8de_82e5ab0806c9.slice/crio-909089ac851bf5fc192469d895ad71f87f3c82146f2e1ea1d6b5d3a3f03c5635 WatchSource:0}: Error finding container 909089ac851bf5fc192469d895ad71f87f3c82146f2e1ea1d6b5d3a3f03c5635: Status 404 returned error can't find the container with id 909089ac851bf5fc192469d895ad71f87f3c82146f2e1ea1d6b5d3a3f03c5635
Sep 11 21:44:59.754350 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.754309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" event={"ID":"48558443-fa02-4878-b8de-82e5ab0806c9","Type":"ContainerStarted","Data":"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"}
Sep 11 21:44:59.754528 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.754364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" event={"ID":"48558443-fa02-4878-b8de-82e5ab0806c9","Type":"ContainerStarted","Data":"909089ac851bf5fc192469d895ad71f87f3c82146f2e1ea1d6b5d3a3f03c5635"}
Sep 11 21:44:59.771766 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:44:59.771720    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" podStartSLOduration=1.771705945 podStartE2EDuration="1.771705945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:59.771109782 +0000 UTC m=+1647.893441701" watchObservedRunningTime="2026-09-11 21:44:59.771705945 +0000 UTC m=+1647.894037910"
Sep 11 21:45:01.393644 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:01.393609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:45:09.197810 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:09.197775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:09.197810 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:09.197823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:09.210182 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:09.210154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:09.816570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:09.816544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:15.492658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:15.492623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:45:24.589292 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:24.589258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:45:29.684989 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:29.684957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:45:30.596731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:30.596700    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:45:34.586246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:34.586212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:45:44.703783 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.703720    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" podUID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" containerName="main" containerID="cri-o://9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae" gracePeriod=45
Sep 11 21:45:44.945056 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.945034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:45:44.973636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.973563    2580 generic.go:356] "Generic (PLEG): container finished" podID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" containerID="9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae" exitCode=0
Sep 11 21:45:44.973636 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.973601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"
Sep 11 21:45:44.973792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.973640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" event={"ID":"e933b1e3-92e2-4899-b4a4-0b535a823a3a","Type":"ContainerDied","Data":"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"}
Sep 11 21:45:44.973792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.973682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj" event={"ID":"e933b1e3-92e2-4899-b4a4-0b535a823a3a","Type":"ContainerDied","Data":"55513a01e9619286c977fcd2aa0009866824b43d7e0ecf60e927c39af330b17b"}
Sep 11 21:45:44.973792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.973699    2580 scope.go:117] "RemoveContainer" containerID="9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"
Sep 11 21:45:44.983119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.983103    2580 scope.go:117] "RemoveContainer" containerID="9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"
Sep 11 21:45:44.983412 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:45:44.983390    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae\": container with ID starting with 9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae not found: ID does not exist" containerID="9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"
Sep 11 21:45:44.983472 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:44.983427    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae"} err="failed to get container status \"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae\": rpc error: code = NotFound desc = could not find container \"9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae\": container with ID starting with 9f8511a551f560658883e2705053399297bf005583fbccc56f2aef803f21a7ae not found: ID does not exist"
Sep 11 21:45:45.049358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049446 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2rm5\" (UniqueName: \"kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049446 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049446 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm\") pod \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\" (UID: \"e933b1e3-92e2-4899-b4a4-0b535a823a3a\") "
Sep 11 21:45:45.049681 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049603    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home" (OuterVolumeSpecName: "home") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.049742 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.049796 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049757    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.049870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049852    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.049925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049873    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.049925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.049882    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.051634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.051604    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:45:45.051634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.051612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5" (OuterVolumeSpecName: "kube-api-access-q2rm5") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "kube-api-access-q2rm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:45.051760 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.051663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm" (OuterVolumeSpecName: "dshm") pod "e933b1e3-92e2-4899-b4a4-0b535a823a3a" (UID: "e933b1e3-92e2-4899-b4a4-0b535a823a3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.150798 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.150778    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2rm5\" (UniqueName: \"kubernetes.io/projected/e933b1e3-92e2-4899-b4a4-0b535a823a3a-kube-api-access-q2rm5\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.150798 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.150799    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e933b1e3-92e2-4899-b4a4-0b535a823a3a-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.150920 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.150808    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e933b1e3-92e2-4899-b4a4-0b535a823a3a-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:45.293924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.293904    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:45:45.299600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.299577    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c97420/rate-test-model-c97420-kserve-6d6cf96567-9fklj"]
Sep 11 21:45:45.616002 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.615902    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" podUID="48558443-fa02-4878-b8de-82e5ab0806c9" containerName="main" containerID="cri-o://3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2" gracePeriod=45
Sep 11 21:45:45.863827 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.863802    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:45.960125 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960125 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9flm2\" (UniqueName: \"kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960348 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache\") pod \"48558443-fa02-4878-b8de-82e5ab0806c9\" (UID: \"48558443-fa02-4878-b8de-82e5ab0806c9\") "
Sep 11 21:45:45.960575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.960575 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home" (OuterVolumeSpecName: "home") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.960660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.960619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.962576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.962542    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:45:45.962834 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.962814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm" (OuterVolumeSpecName: "dshm") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:45:45.962925 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.962851    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2" (OuterVolumeSpecName: "kube-api-access-9flm2") pod "48558443-fa02-4878-b8de-82e5ab0806c9" (UID: "48558443-fa02-4878-b8de-82e5ab0806c9"). InnerVolumeSpecName "kube-api-access-9flm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:45:45.979693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.979668    2580 generic.go:356] "Generic (PLEG): container finished" podID="48558443-fa02-4878-b8de-82e5ab0806c9" containerID="3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2" exitCode=0
Sep 11 21:45:45.979776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.979711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"
Sep 11 21:45:45.979816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.979710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" event={"ID":"48558443-fa02-4878-b8de-82e5ab0806c9","Type":"ContainerDied","Data":"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"}
Sep 11 21:45:45.979859 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.979839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl" event={"ID":"48558443-fa02-4878-b8de-82e5ab0806c9","Type":"ContainerDied","Data":"909089ac851bf5fc192469d895ad71f87f3c82146f2e1ea1d6b5d3a3f03c5635"}
Sep 11 21:45:45.979894 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.979874    2580 scope.go:117] "RemoveContainer" containerID="3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"
Sep 11 21:45:45.988841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.988825    2580 scope.go:117] "RemoveContainer" containerID="3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"
Sep 11 21:45:45.989080 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:45:45.989064    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2\": container with ID starting with 3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2 not found: ID does not exist" containerID="3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"
Sep 11 21:45:45.989141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:45.989091    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2"} err="failed to get container status \"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2\": rpc error: code = NotFound desc = could not find container \"3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2\": container with ID starting with 3a190c2bce6793541e946d1e8b953ac0017afaa9c5dc8038317d58d6bfc066b2 not found: ID does not exist"
Sep 11 21:45:46.000616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.000590    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:45:46.005911 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.005890    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e0c211/rate-test-model-e0c211-kserve-f6b4675d7-rh6xl"]
Sep 11 21:45:46.061261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061234    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.061261 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061256    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48558443-fa02-4878-b8de-82e5ab0806c9-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.061392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061269    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flm2\" (UniqueName: \"kubernetes.io/projected/48558443-fa02-4878-b8de-82e5ab0806c9-kube-api-access-9flm2\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.061392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061282    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.061392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061294    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.061392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.061304    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48558443-fa02-4878-b8de-82e5ab0806c9-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:45:46.383725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.383695    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48558443-fa02-4878-b8de-82e5ab0806c9" path="/var/lib/kubelet/pods/48558443-fa02-4878-b8de-82e5ab0806c9/volumes"
Sep 11 21:45:46.384076 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:46.384064    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" path="/var/lib/kubelet/pods/e933b1e3-92e2-4899-b4a4-0b535a823a3a/volumes"
Sep 11 21:45:47.246578 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:45:47.246549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:46:00.866266 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866179    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:46:00.866739 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866713    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48558443-fa02-4878-b8de-82e5ab0806c9" containerName="main"
Sep 11 21:46:00.866739 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866731    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="48558443-fa02-4878-b8de-82e5ab0806c9" containerName="main"
Sep 11 21:46:00.866853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866752    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" containerName="main"
Sep 11 21:46:00.866853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866760    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" containerName="main"
Sep 11 21:46:00.866948 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866852    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e933b1e3-92e2-4899-b4a4-0b535a823a3a" containerName="main"
Sep 11 21:46:00.866948 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.866868    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="48558443-fa02-4878-b8de-82e5ab0806c9" containerName="main"
Sep 11 21:46:00.876228 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.876204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.878645 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.878623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-213a70-openshift-default-dockercfg-gzh9b\""
Sep 11 21:46:00.882311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.882280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:46:00.971109 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971408 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971408 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971408 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:00.971513 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:00.971415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2md\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072555 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072555 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.072933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2md\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.072970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073038 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.073012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.073165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073293 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.073250    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.073347 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.073324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.075382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.075361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.075531 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.075496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.086225 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.086204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.086892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.086877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2md\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md\") pod \"e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") " pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.187254 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.187198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:01.493469 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.493446    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:46:01.495308 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:46:01.495267    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42002994_edde_4d37_ab34_b600a59b9e59.slice/crio-b5f77635407ed8e0f761fb4d5a534c867b6e5ef41fb464f27621b44ac8ed860e WatchSource:0}: Error finding container b5f77635407ed8e0f761fb4d5a534c867b6e5ef41fb464f27621b44ac8ed860e: Status 404 returned error can't find the container with id b5f77635407ed8e0f761fb4d5a534c867b6e5ef41fb464f27621b44ac8ed860e
Sep 11 21:46:01.497397 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.497365    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:01.497462 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.497439    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:01.497515 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:01.497468    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:02.052777 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:02.052742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerStarted","Data":"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"}
Sep 11 21:46:02.053193 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:02.052801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerStarted","Data":"b5f77635407ed8e0f761fb4d5a534c867b6e5ef41fb464f27621b44ac8ed860e"}
Sep 11 21:46:02.089145 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:02.089100    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podStartSLOduration=2.089084069 podStartE2EDuration="2.089084069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:46:02.088211475 +0000 UTC m=+1710.210543396" watchObservedRunningTime="2026-09-11 21:46:02.089084069 +0000 UTC m=+1710.211415988"
Sep 11 21:46:02.188196 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:02.188162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:02.193128 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:02.193104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:03.057935 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:03.057897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:03.058737 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:03.058719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:07.461615 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.461579    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:46:07.465759 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.465720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.468466 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.468445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-213a70-serving-cert\""
Sep 11 21:46:07.477910 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.477887    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:46:07.523424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.523393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.523562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.523439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.523562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.523466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.523562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.523548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khctx\" (UniqueName: \"kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.624460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.624431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.624616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.624487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khctx\" (UniqueName: \"kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.624616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.624542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.624616 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.624574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.625185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.625163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.626912 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.626891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.627031 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.626919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.632763 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.632742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khctx\" (UniqueName: \"kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx\") pod \"maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:07.776479 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:07.776454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:08.055895 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:08.055862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:46:08.058067 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:46:08.058042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19443a7_b400_44c4_9c9a_4c8e192c066d.slice/crio-c1243f931716af4cba69fedfd1c54fef887717ed5898c4069ed1158eac41866e WatchSource:0}: Error finding container c1243f931716af4cba69fedfd1c54fef887717ed5898c4069ed1158eac41866e: Status 404 returned error can't find the container with id c1243f931716af4cba69fedfd1c54fef887717ed5898c4069ed1158eac41866e
Sep 11 21:46:08.059488 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:08.059471    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:46:08.082478 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:08.082448    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" event={"ID":"e19443a7-b400-44c4-9c9a-4c8e192c066d","Type":"ContainerStarted","Data":"c1243f931716af4cba69fedfd1c54fef887717ed5898c4069ed1158eac41866e"}
Sep 11 21:46:09.089528 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:09.089470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" event={"ID":"e19443a7-b400-44c4-9c9a-4c8e192c066d","Type":"ContainerStarted","Data":"c82628d9025171e13bce83ad39d00ed1ffa16f2ea1162cf11e016f3967fa7993"}
Sep 11 21:46:09.089926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:09.089556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:09.110359 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:09.110314    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" podStartSLOduration=1.782255967 podStartE2EDuration="2.110300109s" totalImagesPullingTime="328.044142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:46:09.10944129 +0000 UTC m=+1717.231773212" watchObservedRunningTime="2026-09-11 21:46:09.110300109 +0000 UTC m=+1717.232632028"
Sep 11 21:46:11.233522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:11.227718    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:46:15.100015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:15.099986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:46:22.155214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.155177    2580 generic.go:356] "Generic (PLEG): container finished" podID="42002994-edde-4d37-ab34-b600a59b9e59" containerID="91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9" exitCode=0
Sep 11 21:46:22.155754 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.155256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerDied","Data":"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"}
Sep 11 21:46:22.155826 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.155772    2580 scope.go:117] "RemoveContainer" containerID="91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"
Sep 11 21:46:22.156870 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.156839    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:22.156969 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.156940    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:22.157030 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:22.157000    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:46:23.163342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:23.163315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerStarted","Data":"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"}
Sep 11 21:46:23.188369 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:23.188343    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:23.188549 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:23.188529    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 21:46:23.188593 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:23.188573    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 21:46:24.187745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:24.187712    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 21:46:24.188108 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:24.187765    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 21:46:25.188089 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:25.188054    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 21:46:25.188566 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:25.188114    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 21:46:26.187833 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:26.187799    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 21:46:26.188026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:26.187857    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 21:46:27.188247 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:27.188218    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 21:46:27.188624 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:27.188280    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 21:46:28.191630 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:28.191605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:28.192018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:28.191857    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:28.192444 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:28.192425    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:46:54.488755 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.488720    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:46:54.492207 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.492188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.494991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.494976    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a6078\"/\"openshift-service-ca.crt\""
Sep 11 21:46:54.495853 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.495832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a6078\"/\"kube-root-ca.crt\""
Sep 11 21:46:54.495944 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.495862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a6078\"/\"ipp-route-9a6078-kserve-self-signed-certs\""
Sep 11 21:46:54.495944 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.495884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a6078\"/\"default-dockercfg-fjw8w\""
Sep 11 21:46:54.503883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.503862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:46:54.591288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.591433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.591433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bpj\" (UniqueName: \"kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.591433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.591640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.591640 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.591527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.692950 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.692925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.692950 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.692952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.692983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.692999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.693022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bpj\" (UniqueName: \"kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.693053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.693387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.693416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.693471 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.693433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.695338 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.695322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.695626 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.695611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.701813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.701786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bpj\" (UniqueName: \"kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj\") pod \"ipp-route-9a6078-kserve-56598b859-xbl5n\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") " pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:54.801433 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:54.801364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:46:55.073295 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:55.073272    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:46:55.075003 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:46:55.074976    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8df17239_e2fa_4082_ad22_79e72d186d4f.slice/crio-e4c3d55718caa72fc1d1426a0357eff4c23d0ef6b5747f0a587b762396d3862a WatchSource:0}: Error finding container e4c3d55718caa72fc1d1426a0357eff4c23d0ef6b5747f0a587b762396d3862a: Status 404 returned error can't find the container with id e4c3d55718caa72fc1d1426a0357eff4c23d0ef6b5747f0a587b762396d3862a
Sep 11 21:46:55.311600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:55.311561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" event={"ID":"8df17239-e2fa-4082-ad22-79e72d186d4f","Type":"ContainerStarted","Data":"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"}
Sep 11 21:46:55.311792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:55.311609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" event={"ID":"8df17239-e2fa-4082-ad22-79e72d186d4f","Type":"ContainerStarted","Data":"e4c3d55718caa72fc1d1426a0357eff4c23d0ef6b5747f0a587b762396d3862a"}
Sep 11 21:46:55.331127 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:55.331014    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" podStartSLOduration=1.330998961 podStartE2EDuration="1.330998961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:46:55.329098868 +0000 UTC m=+1763.451430813" watchObservedRunningTime="2026-09-11 21:46:55.330998961 +0000 UTC m=+1763.453330911"
Sep 11 21:46:58.192715 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:46:58.192678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:47:04.801849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:04.801819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:47:04.801849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:04.801859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:47:04.813888 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:04.813866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:47:05.376724 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:05.376697    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:47:19.013490 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:19.013457    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:47:26.341382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.341308    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:47:26.344650 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.344632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.347170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.347144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-13e3ee-openshift-default-dockercfg-wx2jv\""
Sep 11 21:47:26.355546 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.355524    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:47:26.447774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.447774 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z9l\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.447998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.448048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.448018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548761 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z9l\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.548924 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.548984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549356 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549412 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549481 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549481 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.549627 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.549609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.551356 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.551330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.551463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.551445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.560424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.560406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.573585 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.573560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z9l\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l\") pod \"e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") " pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.656567 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.656492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:26.933211 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.933168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:47:26.941797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.941756    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:26.942003 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.941983    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:26.942102 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:26.942035    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:27.474914 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:27.474880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerStarted","Data":"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"}
Sep 11 21:47:27.475288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:27.474931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerStarted","Data":"c318a92a7b61ff202a383801c67522539e51c7c2fcef3f3ddc9a0d39c31a5013"}
Sep 11 21:47:27.497817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:27.497772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podStartSLOduration=1.49775691 podStartE2EDuration="1.49775691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:27.495935672 +0000 UTC m=+1795.618267590" watchObservedRunningTime="2026-09-11 21:47:27.49775691 +0000 UTC m=+1795.620088886"
Sep 11 21:47:27.657009 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:27.656968    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:27.662044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:27.662019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:28.479731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:28.479707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:28.480639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:28.480615    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:33.014565 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.014531    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:47:33.018671 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.018649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.021386 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.021363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-13e3ee-serving-cert\""
Sep 11 21:47:33.031787 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.031766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:47:33.101814 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.101785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.101920 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.101832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.101920 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.101910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.102043 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.102022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp46s\" (UniqueName: \"kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.203124 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.203088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp46s\" (UniqueName: \"kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.203298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.203141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.203298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.203199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.203298 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.203282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.203880 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.203855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.205762 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.205731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.205877 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.205780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.226026 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.226005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp46s\" (UniqueName: \"kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s\") pod \"maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.329410 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.329339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:33.621776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:33.621754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:47:33.623458 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:47:33.623430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2158088c_4aa9_4965_8ed7_dde96ffec92a.slice/crio-c9c38c5546d0f75739680fb102a25fa4be023f844417b505c5b252b26e8a18c5 WatchSource:0}: Error finding container c9c38c5546d0f75739680fb102a25fa4be023f844417b505c5b252b26e8a18c5: Status 404 returned error can't find the container with id c9c38c5546d0f75739680fb102a25fa4be023f844417b505c5b252b26e8a18c5
Sep 11 21:47:34.511414 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:34.511381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" event={"ID":"2158088c-4aa9-4965-8ed7-dde96ffec92a","Type":"ContainerStarted","Data":"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"}
Sep 11 21:47:34.511776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:34.511437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" event={"ID":"2158088c-4aa9-4965-8ed7-dde96ffec92a","Type":"ContainerStarted","Data":"c9c38c5546d0f75739680fb102a25fa4be023f844417b505c5b252b26e8a18c5"}
Sep 11 21:47:34.511776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:34.511481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:34.531061 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:34.531007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" podStartSLOduration=2.042113811 podStartE2EDuration="2.530990535s" totalImagesPullingTime="488.876724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:34.529281599 +0000 UTC m=+1802.651613531" watchObservedRunningTime="2026-09-11 21:47:34.530990535 +0000 UTC m=+1802.653322651"
Sep 11 21:47:37.796277 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:37.796236    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:47:40.522677 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:40.522654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:47:48.591858 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.591824    2580 generic.go:356] "Generic (PLEG): container finished" podID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerID="0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570" exitCode=0
Sep 11 21:47:48.592278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.591895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerDied","Data":"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"}
Sep 11 21:47:48.592278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.592223    2580 scope.go:117] "RemoveContainer" containerID="0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"
Sep 11 21:47:48.593234 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.593208    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:48.593284 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.593274    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:48.593318 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:48.593303    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 11 21:47:49.598559 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:49.598525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerStarted","Data":"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"}
Sep 11 21:47:49.657474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:49.657437    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:49.657818 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:49.657788    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:49.657930 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:49.657839    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:50.657807 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:50.657774    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:50.658305 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:50.657832    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:51.657852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:51.657814    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:51.658238 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:51.657877    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:52.657437 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:52.657402    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:52.657641 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:52.657480    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:53.657573 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:53.657541    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:53.657952 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:53.657602    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:54.657232 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:54.657199    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 21:47:54.657405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:54.657257    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 21:47:55.660704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:55.660677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:55.661169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:55.661033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:47:55.661730 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:47:55.661715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:48:11.318694 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.318660    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:48:11.319279 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.319027    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" podUID="2158088c-4aa9-4965-8ed7-dde96ffec92a" containerName="maas-api" containerID="cri-o://be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb" gracePeriod=30
Sep 11 21:48:11.569590 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.569539    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:48:11.645552 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls\") pod \"2158088c-4aa9-4965-8ed7-dde96ffec92a\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") "
Sep 11 21:48:11.645703 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645594    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca\") pod \"2158088c-4aa9-4965-8ed7-dde96ffec92a\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") "
Sep 11 21:48:11.645703 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp46s\" (UniqueName: \"kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s\") pod \"2158088c-4aa9-4965-8ed7-dde96ffec92a\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") "
Sep 11 21:48:11.645794 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs\") pod \"2158088c-4aa9-4965-8ed7-dde96ffec92a\" (UID: \"2158088c-4aa9-4965-8ed7-dde96ffec92a\") "
Sep 11 21:48:11.645916 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645888    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2158088c-4aa9-4965-8ed7-dde96ffec92a" (UID: "2158088c-4aa9-4965-8ed7-dde96ffec92a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:11.646018 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.645993    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2158088c-4aa9-4965-8ed7-dde96ffec92a-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:11.647844 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.647817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2158088c-4aa9-4965-8ed7-dde96ffec92a" (UID: "2158088c-4aa9-4965-8ed7-dde96ffec92a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:11.648324 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.648303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2158088c-4aa9-4965-8ed7-dde96ffec92a" (UID: "2158088c-4aa9-4965-8ed7-dde96ffec92a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:11.648405 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.648379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s" (OuterVolumeSpecName: "kube-api-access-pp46s") pod "2158088c-4aa9-4965-8ed7-dde96ffec92a" (UID: "2158088c-4aa9-4965-8ed7-dde96ffec92a"). InnerVolumeSpecName "kube-api-access-pp46s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:11.701065 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.701035    2580 generic.go:356] "Generic (PLEG): container finished" podID="2158088c-4aa9-4965-8ed7-dde96ffec92a" containerID="be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb" exitCode=0
Sep 11 21:48:11.701170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.701073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"
Sep 11 21:48:11.701170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.701109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" event={"ID":"2158088c-4aa9-4965-8ed7-dde96ffec92a","Type":"ContainerDied","Data":"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"}
Sep 11 21:48:11.701170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.701154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z" event={"ID":"2158088c-4aa9-4965-8ed7-dde96ffec92a","Type":"ContainerDied","Data":"c9c38c5546d0f75739680fb102a25fa4be023f844417b505c5b252b26e8a18c5"}
Sep 11 21:48:11.701271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.701178    2580 scope.go:117] "RemoveContainer" containerID="be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"
Sep 11 21:48:11.711931 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.711910    2580 scope.go:117] "RemoveContainer" containerID="be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"
Sep 11 21:48:11.712214 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:48:11.712196    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb\": container with ID starting with be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb not found: ID does not exist" containerID="be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"
Sep 11 21:48:11.712283 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.712222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb"} err="failed to get container status \"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb\": rpc error: code = NotFound desc = could not find container \"be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb\": container with ID starting with be27c67eaa6f73d439af3b195f4225958a10520ccd61916e890d191009d919eb not found: ID does not exist"
Sep 11 21:48:11.724595 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.724571    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:48:11.731524 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.728536    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-13e3ee-d488457b7-7kf9z"]
Sep 11 21:48:11.746706 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.746683    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp46s\" (UniqueName: \"kubernetes.io/projected/2158088c-4aa9-4965-8ed7-dde96ffec92a-kube-api-access-pp46s\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:11.746817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.746707    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:11.746817 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:11.746719    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2158088c-4aa9-4965-8ed7-dde96ffec92a-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:12.383933 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:12.383904    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2158088c-4aa9-4965-8ed7-dde96ffec92a" path="/var/lib/kubelet/pods/2158088c-4aa9-4965-8ed7-dde96ffec92a/volumes"
Sep 11 21:48:15.893333 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:15.893290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:48:18.422445 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:18.422414    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:48:18.422879 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:18.422686    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy" containerID="cri-o://a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900" gracePeriod=30
Sep 11 21:48:23.676799 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.676773    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:48:23.745727 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.745883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.745883 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26z9l\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.745974    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:23.746010 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data\") pod \"da8ad276-706b-4fe8-8632-e2e0969d92f1\" (UID: \"da8ad276-706b-4fe8-8632-e2e0969d92f1\") "
Sep 11 21:48:23.746276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:23.746276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:23.746484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:23.746484 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:23.746764 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746617    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da8ad276-706b-4fe8-8632-e2e0969d92f1-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.746764 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746635    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.746764 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746649    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.746764 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.746662    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.748132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.748102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l" (OuterVolumeSpecName: "kube-api-access-26z9l") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "kube-api-access-26z9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:23.748132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.748116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:23.748262 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.748177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:23.748262 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.748188    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da8ad276-706b-4fe8-8632-e2e0969d92f1" (UID: "da8ad276-706b-4fe8-8632-e2e0969d92f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:48:23.763611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.763588    2580 generic.go:356] "Generic (PLEG): container finished" podID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerID="a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900" exitCode=0
Sep 11 21:48:23.763707 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.763631    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"
Sep 11 21:48:23.763707 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.763673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerDied","Data":"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"}
Sep 11 21:48:23.763797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.763730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh" event={"ID":"da8ad276-706b-4fe8-8632-e2e0969d92f1","Type":"ContainerDied","Data":"c318a92a7b61ff202a383801c67522539e51c7c2fcef3f3ddc9a0d39c31a5013"}
Sep 11 21:48:23.763797 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.763757    2580 scope.go:117] "RemoveContainer" containerID="a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"
Sep 11 21:48:23.772779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.772762    2580 scope.go:117] "RemoveContainer" containerID="0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"
Sep 11 21:48:23.780328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.780313    2580 scope.go:117] "RemoveContainer" containerID="a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"
Sep 11 21:48:23.780621 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:48:23.780602    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900\": container with ID starting with a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900 not found: ID does not exist" containerID="a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"
Sep 11 21:48:23.780679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.780630    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900"} err="failed to get container status \"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900\": rpc error: code = NotFound desc = could not find container \"a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900\": container with ID starting with a122fe5ee0261dc0ac4aba90ec8ce6f523c481abe0f6440bcadd925ee84fd900 not found: ID does not exist"
Sep 11 21:48:23.780679 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.780646    2580 scope.go:117] "RemoveContainer" containerID="0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"
Sep 11 21:48:23.780896 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:48:23.780879    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570\": container with ID starting with 0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570 not found: ID does not exist" containerID="0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"
Sep 11 21:48:23.780939 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.780902    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570"} err="failed to get container status \"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570\": rpc error: code = NotFound desc = could not find container \"0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570\": container with ID starting with 0776b3973e1386c5f7234e88f8c079d064672d4fb231d69436a545cbe8aa3570 not found: ID does not exist"
Sep 11 21:48:23.792355 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.792330    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:48:23.800241 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.800223    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-13e3ee-openshift-default-7cdb84f988-pdfgh"]
Sep 11 21:48:23.847170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.847152    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.847170 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.847170    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.847288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.847178    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26z9l\" (UniqueName: \"kubernetes.io/projected/da8ad276-706b-4fe8-8632-e2e0969d92f1-kube-api-access-26z9l\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.847288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.847188    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:23.847288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:23.847196    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da8ad276-706b-4fe8-8632-e2e0969d92f1-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:24.383147 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:24.383117    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" path="/var/lib/kubelet/pods/da8ad276-706b-4fe8-8632-e2e0969d92f1/volumes"
Sep 11 21:48:35.280788 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:35.280753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:48:43.312954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:43.312915    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:48:47.594276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:47.594234    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:48:58.332111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.332015    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" podUID="8df17239-e2fa-4082-ad22-79e72d186d4f" containerName="main" containerID="cri-o://d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5" gracePeriod=45
Sep 11 21:48:58.574937 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.574908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:48:58.618666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.618666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.618857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.618857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8bpj\" (UniqueName: \"kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.618857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home" (OuterVolumeSpecName: "home") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:58.618857 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618816    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.619062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir\") pod \"8df17239-e2fa-4082-ad22-79e72d186d4f\" (UID: \"8df17239-e2fa-4082-ad22-79e72d186d4f\") "
Sep 11 21:48:58.619062 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.618992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:58.619202 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.619177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:58.619267 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.619250    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.619335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.619272    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.620913 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.620891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:58.621232 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.621209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj" (OuterVolumeSpecName: "kube-api-access-t8bpj") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "kube-api-access-t8bpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:58.621341 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.621233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm" (OuterVolumeSpecName: "dshm") pod "8df17239-e2fa-4082-ad22-79e72d186d4f" (UID: "8df17239-e2fa-4082-ad22-79e72d186d4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:48:58.720522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.720480    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8df17239-e2fa-4082-ad22-79e72d186d4f-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.720522 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.720522    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bpj\" (UniqueName: \"kubernetes.io/projected/8df17239-e2fa-4082-ad22-79e72d186d4f-kube-api-access-t8bpj\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.720658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.720532    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.720658 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.720540    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8df17239-e2fa-4082-ad22-79e72d186d4f-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:48:58.928637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.928571    2580 generic.go:356] "Generic (PLEG): container finished" podID="8df17239-e2fa-4082-ad22-79e72d186d4f" containerID="d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5" exitCode=0
Sep 11 21:48:58.928637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.928609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" event={"ID":"8df17239-e2fa-4082-ad22-79e72d186d4f","Type":"ContainerDied","Data":"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"}
Sep 11 21:48:58.928637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.928618    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"
Sep 11 21:48:58.928637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.928636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n" event={"ID":"8df17239-e2fa-4082-ad22-79e72d186d4f","Type":"ContainerDied","Data":"e4c3d55718caa72fc1d1426a0357eff4c23d0ef6b5747f0a587b762396d3862a"}
Sep 11 21:48:58.928892 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.928652    2580 scope.go:117] "RemoveContainer" containerID="d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"
Sep 11 21:48:58.937623 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.937602    2580 scope.go:117] "RemoveContainer" containerID="d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"
Sep 11 21:48:58.937909 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:48:58.937891    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5\": container with ID starting with d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5 not found: ID does not exist" containerID="d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"
Sep 11 21:48:58.937963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.937919    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5"} err="failed to get container status \"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5\": rpc error: code = NotFound desc = could not find container \"d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5\": container with ID starting with d4a3f9ec0d18573bd9d540e3cbb3b90516885ffa8d9bc4f794865644ae8950c5 not found: ID does not exist"
Sep 11 21:48:58.949752 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.949732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:48:58.953906 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:48:58.953887    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-9a6078/ipp-route-9a6078-kserve-56598b859-xbl5n"]
Sep 11 21:49:00.383407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:00.383372    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df17239-e2fa-4082-ad22-79e72d186d4f" path="/var/lib/kubelet/pods/8df17239-e2fa-4082-ad22-79e72d186d4f/volumes"
Sep 11 21:49:42.901288 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:42.901251    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:49:42.902472 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:42.901475    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" podUID="e19443a7-b400-44c4-9c9a-4c8e192c066d" containerName="maas-api" containerID="cri-o://c82628d9025171e13bce83ad39d00ed1ffa16f2ea1162cf11e016f3967fa7993" gracePeriod=30
Sep 11 21:49:43.132222 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.132189    2580 generic.go:356] "Generic (PLEG): container finished" podID="e19443a7-b400-44c4-9c9a-4c8e192c066d" containerID="c82628d9025171e13bce83ad39d00ed1ffa16f2ea1162cf11e016f3967fa7993" exitCode=0
Sep 11 21:49:43.132400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.132227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" event={"ID":"e19443a7-b400-44c4-9c9a-4c8e192c066d","Type":"ContainerDied","Data":"c82628d9025171e13bce83ad39d00ed1ffa16f2ea1162cf11e016f3967fa7993"}
Sep 11 21:49:43.156776 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.156726    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:49:43.250431 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.250401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs\") pod \"e19443a7-b400-44c4-9c9a-4c8e192c066d\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") "
Sep 11 21:49:43.250614 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.250474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca\") pod \"e19443a7-b400-44c4-9c9a-4c8e192c066d\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") "
Sep 11 21:49:43.250614 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.250544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls\") pod \"e19443a7-b400-44c4-9c9a-4c8e192c066d\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") "
Sep 11 21:49:43.250614 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.250609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khctx\" (UniqueName: \"kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx\") pod \"e19443a7-b400-44c4-9c9a-4c8e192c066d\" (UID: \"e19443a7-b400-44c4-9c9a-4c8e192c066d\") "
Sep 11 21:49:43.250897 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.250871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e19443a7-b400-44c4-9c9a-4c8e192c066d" (UID: "e19443a7-b400-44c4-9c9a-4c8e192c066d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:49:43.252698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.252671    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e19443a7-b400-44c4-9c9a-4c8e192c066d" (UID: "e19443a7-b400-44c4-9c9a-4c8e192c066d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:49:43.252802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.252775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx" (OuterVolumeSpecName: "kube-api-access-khctx") pod "e19443a7-b400-44c4-9c9a-4c8e192c066d" (UID: "e19443a7-b400-44c4-9c9a-4c8e192c066d"). InnerVolumeSpecName "kube-api-access-khctx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:49:43.252851 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.252805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e19443a7-b400-44c4-9c9a-4c8e192c066d" (UID: "e19443a7-b400-44c4-9c9a-4c8e192c066d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:49:43.351274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.351241    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:43.351274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.351267    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e19443a7-b400-44c4-9c9a-4c8e192c066d-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:43.351274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.351277    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e19443a7-b400-44c4-9c9a-4c8e192c066d-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:43.351274 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:43.351285    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khctx\" (UniqueName: \"kubernetes.io/projected/e19443a7-b400-44c4-9c9a-4c8e192c066d-kube-api-access-khctx\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:44.138631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.138601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"
Sep 11 21:49:44.138631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.138615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj" event={"ID":"e19443a7-b400-44c4-9c9a-4c8e192c066d","Type":"ContainerDied","Data":"c1243f931716af4cba69fedfd1c54fef887717ed5898c4069ed1158eac41866e"}
Sep 11 21:49:44.139089 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.138670    2580 scope.go:117] "RemoveContainer" containerID="c82628d9025171e13bce83ad39d00ed1ffa16f2ea1162cf11e016f3967fa7993"
Sep 11 21:49:44.160610 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.160581    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:49:44.165125 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.165105    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-213a70-6784cb8549-s8sxj"]
Sep 11 21:49:44.383496 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:44.383469    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19443a7-b400-44c4-9c9a-4c8e192c066d" path="/var/lib/kubelet/pods/e19443a7-b400-44c4-9c9a-4c8e192c066d/volumes"
Sep 11 21:49:46.048532 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:46.048485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:49:49.726898 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:49.726863    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:49:49.727354 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:49.727081    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy" containerID="cri-o://073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f" gracePeriod=30
Sep 11 21:49:54.966048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:54.966018    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:49:55.051419 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051419 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz2md\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051633 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051634    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:49:55.051941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert\") pod \"42002994-edde-4d37-ab34-b600a59b9e59\" (UID: \"42002994-edde-4d37-ab34-b600a59b9e59\") "
Sep 11 21:49:55.051941 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:49:55.052091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051959    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.052091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051977    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-credential-socket\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.052091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.051989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:49:55.052091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.052014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:49:55.052475 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.052174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data" (OuterVolumeSpecName: "istio-data") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:49:55.053756 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.053736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token" (OuterVolumeSpecName: "istio-token") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:49:55.053849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.053788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md" (OuterVolumeSpecName: "kube-api-access-cz2md") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "kube-api-access-cz2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:49:55.053849 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.053814    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:49:55.053957 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.053940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42002994-edde-4d37-ab34-b600a59b9e59" (UID: "42002994-edde-4d37-ab34-b600a59b9e59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:49:55.158680 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158655    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-istio-token\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158838 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158824    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-data\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158845    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2md\" (UniqueName: \"kubernetes.io/projected/42002994-edde-4d37-ab34-b600a59b9e59-kube-api-access-cz2md\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158857    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42002994-edde-4d37-ab34-b600a59b9e59-istio-podinfo\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158878 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158867    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-istio-envoy\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158980 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158880    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42002994-edde-4d37-ab34-b600a59b9e59-workload-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.158980 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.158891    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42002994-edde-4d37-ab34-b600a59b9e59-istiod-ca-cert\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:49:55.192608 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.192580    2580 generic.go:356] "Generic (PLEG): container finished" podID="42002994-edde-4d37-ab34-b600a59b9e59" containerID="073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f" exitCode=0
Sep 11 21:49:55.192713 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.192631    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"
Sep 11 21:49:55.192713 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.192646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerDied","Data":"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"}
Sep 11 21:49:55.192829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.192728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm" event={"ID":"42002994-edde-4d37-ab34-b600a59b9e59","Type":"ContainerDied","Data":"b5f77635407ed8e0f761fb4d5a534c867b6e5ef41fb464f27621b44ac8ed860e"}
Sep 11 21:49:55.192829 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.192760    2580 scope.go:117] "RemoveContainer" containerID="073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"
Sep 11 21:49:55.201227 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.201211    2580 scope.go:117] "RemoveContainer" containerID="91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"
Sep 11 21:49:55.208687 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.208670    2580 scope.go:117] "RemoveContainer" containerID="073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"
Sep 11 21:49:55.208939 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:49:55.208922    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f\": container with ID starting with 073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f not found: ID does not exist" containerID="073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"
Sep 11 21:49:55.208983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.208948    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f"} err="failed to get container status \"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f\": rpc error: code = NotFound desc = could not find container \"073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f\": container with ID starting with 073d19199e13d398c191fa42b0698183ec0e6ae113b1b0ef6c72ed4dc02ad30f not found: ID does not exist"
Sep 11 21:49:55.208983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.208963    2580 scope.go:117] "RemoveContainer" containerID="91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"
Sep 11 21:49:55.209191 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:49:55.209177    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9\": container with ID starting with 91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9 not found: ID does not exist" containerID="91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"
Sep 11 21:49:55.209236 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.209194    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9"} err="failed to get container status \"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9\": rpc error: code = NotFound desc = could not find container \"91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9\": container with ID starting with 91576687ac2d5a10883d253d81ff69c2dc350a4be689d66081d4587eb54469f9 not found: ID does not exist"
Sep 11 21:49:55.236056 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.236038    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:49:55.246762 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:55.246743    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-213a70-openshift-default-5d7754d488-dkpsm"]
Sep 11 21:49:56.382958 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:49:56.382929    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42002994-edde-4d37-ab34-b600a59b9e59" path="/var/lib/kubelet/pods/42002994-edde-4d37-ab34-b600a59b9e59/volumes"
Sep 11 21:50:00.572963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.572724    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:50:00.572963 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.572938    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" podUID="49f20de6-f34d-4d72-a355-24a344334f6e" containerName="maas-api" containerID="cri-o://5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71" gracePeriod=30
Sep 11 21:50:00.814731 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.814707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:50:00.902017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.901953    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94v4q\" (UniqueName: \"kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q\") pod \"49f20de6-f34d-4d72-a355-24a344334f6e\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") "
Sep 11 21:50:00.902017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.901990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca\") pod \"49f20de6-f34d-4d72-a355-24a344334f6e\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") "
Sep 11 21:50:00.902185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.902025    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") pod \"49f20de6-f34d-4d72-a355-24a344334f6e\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") "
Sep 11 21:50:00.902185 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.902057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs\") pod \"49f20de6-f34d-4d72-a355-24a344334f6e\" (UID: \"49f20de6-f34d-4d72-a355-24a344334f6e\") "
Sep 11 21:50:00.902337 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.902317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49f20de6-f34d-4d72-a355-24a344334f6e" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:00.904263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.904239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49f20de6-f34d-4d72-a355-24a344334f6e" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:00.904375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.904298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49f20de6-f34d-4d72-a355-24a344334f6e" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:00.904375 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:00.904300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q" (OuterVolumeSpecName: "kube-api-access-94v4q") pod "49f20de6-f34d-4d72-a355-24a344334f6e" (UID: "49f20de6-f34d-4d72-a355-24a344334f6e"). InnerVolumeSpecName "kube-api-access-94v4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:50:01.009075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.009037    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f20de6-f34d-4d72-a355-24a344334f6e-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:01.009075 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.009073    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:01.009263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.009096    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49f20de6-f34d-4d72-a355-24a344334f6e-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:01.009263 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.009128    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94v4q\" (UniqueName: \"kubernetes.io/projected/49f20de6-f34d-4d72-a355-24a344334f6e-kube-api-access-94v4q\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:01.220106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.220033    2580 generic.go:356] "Generic (PLEG): container finished" podID="49f20de6-f34d-4d72-a355-24a344334f6e" containerID="5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71" exitCode=0
Sep 11 21:50:01.220106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.220092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"
Sep 11 21:50:01.220271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.220121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" event={"ID":"49f20de6-f34d-4d72-a355-24a344334f6e","Type":"ContainerDied","Data":"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"}
Sep 11 21:50:01.220271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.220206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt" event={"ID":"49f20de6-f34d-4d72-a355-24a344334f6e","Type":"ContainerDied","Data":"9cf38e6f6779f0cf9eaa15e853cebc623b00b9b21f8632cd677ca2d6c0dfd1f4"}
Sep 11 21:50:01.220271 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.220234    2580 scope.go:117] "RemoveContainer" containerID="5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"
Sep 11 21:50:01.229213 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.229192    2580 scope.go:117] "RemoveContainer" containerID="5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"
Sep 11 21:50:01.229571 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:50:01.229486    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71\": container with ID starting with 5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71 not found: ID does not exist" containerID="5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"
Sep 11 21:50:01.229631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.229583    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71"} err="failed to get container status \"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71\": rpc error: code = NotFound desc = could not find container \"5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71\": container with ID starting with 5556bc76a13c6b50fec74b5d28e2406412ab2b33cdec5e4bd628e54bd7984a71 not found: ID does not exist"
Sep 11 21:50:01.242651 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.242627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:50:01.247389 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:01.247365    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c97420-75bb8df768-7lwvt"]
Sep 11 21:50:02.384015 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:02.383982    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f20de6-f34d-4d72-a355-24a344334f6e" path="/var/lib/kubelet/pods/49f20de6-f34d-4d72-a355-24a344334f6e/volumes"
Sep 11 21:50:03.388106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:03.388066    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:50:18.214795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.214760    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:50:18.215693 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.215062    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" podUID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" containerName="maas-api" containerID="cri-o://67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f" gracePeriod=30
Sep 11 21:50:18.463381 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.463359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:50:18.643282 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.643251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca\") pod \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") "
Sep 11 21:50:18.643443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.643318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs\") pod \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") "
Sep 11 21:50:18.643443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.643350    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") pod \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") "
Sep 11 21:50:18.643443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.643385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfstr\" (UniqueName: \"kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr\") pod \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\" (UID: \"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b\") "
Sep 11 21:50:18.643663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.643638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" (UID: "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:18.645556 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.645530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" (UID: "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:18.645666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.645568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr" (OuterVolumeSpecName: "kube-api-access-xfstr") pod "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" (UID: "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b"). InnerVolumeSpecName "kube-api-access-xfstr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:50:18.645709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.645672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" (UID: "ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:18.744346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.744319    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:18.744346 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.744341    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:18.744483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.744350    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfstr\" (UniqueName: \"kubernetes.io/projected/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-kube-api-access-xfstr\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:18.744483 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:18.744360    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:50:19.303490 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.303453    2580 generic.go:356] "Generic (PLEG): container finished" podID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" containerID="67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f" exitCode=0
Sep 11 21:50:19.303884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.303509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"
Sep 11 21:50:19.303884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.303545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" event={"ID":"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b","Type":"ContainerDied","Data":"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"}
Sep 11 21:50:19.303884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.303601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4" event={"ID":"ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b","Type":"ContainerDied","Data":"2e2810fb56cc6a2d138068203919dec0031345845290bb63381d51f30f35c483"}
Sep 11 21:50:19.303884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.303624    2580 scope.go:117] "RemoveContainer" containerID="67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"
Sep 11 21:50:19.312448 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.312429    2580 scope.go:117] "RemoveContainer" containerID="67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"
Sep 11 21:50:19.312750 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:50:19.312732    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f\": container with ID starting with 67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f not found: ID does not exist" containerID="67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"
Sep 11 21:50:19.312808 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.312760    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f"} err="failed to get container status \"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f\": rpc error: code = NotFound desc = could not find container \"67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f\": container with ID starting with 67ca829c1b01b000706f3368643012aa0d0a1b474a5ef944773122ed1036016f not found: ID does not exist"
Sep 11 21:50:19.324856 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.324823    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:50:19.327106 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:19.327089    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e0c211-8444b8f894-9x5g4"]
Sep 11 21:50:20.382779 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:20.382748    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" path="/var/lib/kubelet/pods/ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b/volumes"
Sep 11 21:50:20.800852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:20.800819    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:50:29.400250 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:50:29.400049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:51:51.181091 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:51:51.181058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:16.291828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:16.291791    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:22.893235 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:22.893202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:28.920816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:28.920784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:36.016257 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:36.016222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:51.799094 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:51.799058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:57.788126 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:57.788093    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:52:59.351311 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:59.351277    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:52:59.351785 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:52:59.351492    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" podUID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" containerName="manager" containerID="cri-o://59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619" gracePeriod=10
Sep 11 21:53:01.291655 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.291632    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:53:01.389666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.389596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk8dn\" (UniqueName: \"kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn\") pod \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") "
Sep 11 21:53:01.389666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.389658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume\") pod \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\" (UID: \"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f\") "
Sep 11 21:53:01.389964 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.389942    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" (UID: "bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:01.391663 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.391645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn" (OuterVolumeSpecName: "kube-api-access-sk8dn") pod "bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" (UID: "bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f"). InnerVolumeSpecName "kube-api-access-sk8dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:01.490424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.490398    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8dn\" (UniqueName: \"kubernetes.io/projected/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-kube-api-access-sk8dn\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:53:01.490424 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:01.490421    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f-extensions-socket-volume\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:53:02.009660 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.009630    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" containerID="59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619" exitCode=0
Sep 11 21:53:02.009813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.009669    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"
Sep 11 21:53:02.009813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.009714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" event={"ID":"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f","Type":"ContainerDied","Data":"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"}
Sep 11 21:53:02.009813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.009772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls" event={"ID":"bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f","Type":"ContainerDied","Data":"256b010893247662c64f6dc15ccc805ba343eff98a2f2af661aa81aaed09194b"}
Sep 11 21:53:02.009813 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.009799    2580 scope.go:117] "RemoveContainer" containerID="59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"
Sep 11 21:53:02.018893 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.018876    2580 scope.go:117] "RemoveContainer" containerID="59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"
Sep 11 21:53:02.019186 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:53:02.019162    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619\": container with ID starting with 59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619 not found: ID does not exist" containerID="59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"
Sep 11 21:53:02.019278 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.019191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619"} err="failed to get container status \"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619\": rpc error: code = NotFound desc = could not find container \"59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619\": container with ID starting with 59752d3036f7928316bb706843f821794847bb8897a9c8c1c7b80e8dc6071619 not found: ID does not exist"
Sep 11 21:53:02.030841 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.030820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:53:02.034072 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.034052    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d9dls"]
Sep 11 21:53:02.383742 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:53:02.383669    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" path="/var/lib/kubelet/pods/bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f/volumes"
Sep 11 21:54:00.036276 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"]
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036683    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" containerName="manager"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036702    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" containerName="manager"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036714    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df17239-e2fa-4082-ad22-79e72d186d4f" containerName="main"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df17239-e2fa-4082-ad22-79e72d186d4f" containerName="main"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036736    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036746    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036758    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036766    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036774    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036782    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" containerName="maas-api"
Sep 11 21:54:00.036795 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036799    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2158088c-4aa9-4965-8ed7-dde96ffec92a" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036819    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2158088c-4aa9-4965-8ed7-dde96ffec92a" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f20de6-f34d-4d72-a355-24a344334f6e" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036847    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f20de6-f34d-4d72-a355-24a344334f6e" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036863    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19443a7-b400-44c4-9c9a-4c8e192c066d" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036871    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19443a7-b400-44c4-9c9a-4c8e192c066d" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036965    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036977    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19443a7-b400-44c4-9c9a-4c8e192c066d" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036986    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac31b6e4-acc0-41dd-b4ca-c8f5bb536a6b" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.036994    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.037001    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.037010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2158088c-4aa9-4965-8ed7-dde96ffec92a" containerName="maas-api"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.037018    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc842d0a-b4ea-4cda-9fa9-b01d8d18cd0f" containerName="manager"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.037026    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df17239-e2fa-4082-ad22-79e72d186d4f" containerName="main"
Sep 11 21:54:00.037361 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.037044    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f20de6-f34d-4d72-a355-24a344334f6e" containerName="maas-api"
Sep 11 21:54:00.040243 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.040223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.043377 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.043358    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-chzk9\""
Sep 11 21:54:00.056396 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.056370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"]
Sep 11 21:54:00.199880 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.199851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.199992 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.199904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcj7l\" (UniqueName: \"kubernetes.io/projected/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-kube-api-access-rcj7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.300692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.300637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.300692 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.300676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcj7l\" (UniqueName: \"kubernetes.io/projected/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-kube-api-access-rcj7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.301039 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.301020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.309907 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.309886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcj7l\" (UniqueName: \"kubernetes.io/projected/3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062-kube-api-access-rcj7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-98jkc\" (UID: \"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.349855 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.349835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:00.630214 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:00.630191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"]
Sep 11 21:54:00.632382 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:54:00.632354    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4a8f74_4c04_4c1f_90e5_95d3e9ea6062.slice/crio-8631b181e711ade0ac897203d3c2061261f0ae86eca5c806abc8ffb772fd27f2 WatchSource:0}: Error finding container 8631b181e711ade0ac897203d3c2061261f0ae86eca5c806abc8ffb772fd27f2: Status 404 returned error can't find the container with id 8631b181e711ade0ac897203d3c2061261f0ae86eca5c806abc8ffb772fd27f2
Sep 11 21:54:01.264369 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:01.264336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc" event={"ID":"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062","Type":"ContainerStarted","Data":"015ef543f337daa56f9b4bfca2fa73173b5d33ce1933c01ae679a3c9966a353c"}
Sep 11 21:54:01.264757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:01.264384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc" event={"ID":"3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062","Type":"ContainerStarted","Data":"8631b181e711ade0ac897203d3c2061261f0ae86eca5c806abc8ffb772fd27f2"}
Sep 11 21:54:01.264757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:01.264436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:01.286225 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:01.286174    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc" podStartSLOduration=1.286160384 podStartE2EDuration="1.286160384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:01.284340062 +0000 UTC m=+2189.406671996" watchObservedRunningTime="2026-09-11 21:54:01.286160384 +0000 UTC m=+2189.408492294"
Sep 11 21:54:12.270667 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:12.270634    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-98jkc"
Sep 11 21:54:22.593367 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:22.593333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:54:30.886490 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:30.886455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:54:33.788234 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.788198    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:54:33.788631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.788510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:33.788631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.788524    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42002994-edde-4d37-ab34-b600a59b9e59" containerName="istio-proxy"
Sep 11 21:54:33.788631 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.788598    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8ad276-706b-4fe8-8632-e2e0969d92f1" containerName="istio-proxy"
Sep 11 21:54:33.791884 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.791866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.794634 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.794608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b4958497-kserve-self-signed-certs\""
Sep 11 21:54:33.800419 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.800395    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:54:33.835486 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.835602 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.835656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wwj\" (UniqueName: \"kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.835656 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.835757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.835811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.835782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.936926 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.936900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wwj\" (UniqueName: \"kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.936932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.936949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.936982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.937012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937047 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.937043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937388 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.937351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937388 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.937373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.937482 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.937422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.939334 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.939315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.939684 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.939664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:33.944642 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:33.944622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wwj\" (UniqueName: \"kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj\") pod \"e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") " pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:34.103600 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:34.103525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:34.375778 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:34.375749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:54:34.378562 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:54:34.378490    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99f9c2e7_4eb2_4450_ae8e_1e01a6b04204.slice/crio-e930935405aebfbbe9cff468a977f7c7116cd7c2e703d507a1d036a2d9222d02 WatchSource:0}: Error finding container e930935405aebfbbe9cff468a977f7c7116cd7c2e703d507a1d036a2d9222d02: Status 404 returned error can't find the container with id e930935405aebfbbe9cff468a977f7c7116cd7c2e703d507a1d036a2d9222d02
Sep 11 21:54:34.406594 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:34.406568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" event={"ID":"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204","Type":"ContainerStarted","Data":"e930935405aebfbbe9cff468a977f7c7116cd7c2e703d507a1d036a2d9222d02"}
Sep 11 21:54:35.412802 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:35.412768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" event={"ID":"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204","Type":"ContainerStarted","Data":"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"}
Sep 11 21:54:35.431191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:35.431129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" podStartSLOduration=2.4311092 podStartE2EDuration="2.4311092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:35.430363802 +0000 UTC m=+2223.552695719" watchObservedRunningTime="2026-09-11 21:54:35.4311092 +0000 UTC m=+2223.553441122"
Sep 11 21:54:36.695301 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.695255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:54:36.699097 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.699076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.703313 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.703291    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b4958497-kserve-self-signed-certs\""
Sep 11 21:54:36.708053 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.708022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:54:36.758780 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.758953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2zp\" (UniqueName: \"kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.758953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.758953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.758953 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.759133 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.758982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860149 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860325 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860325 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860325 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2zp\" (UniqueName: \"kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860647 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860647 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.860757 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.860720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.863027 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.862992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.863395 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.863374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:36.869317 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:36.869295    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2zp\" (UniqueName: \"kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp\") pod \"e2e-same-id-b-b4958497-kserve-867749f659-hrpsp\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") " pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:37.020362 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:37.020325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:37.293584 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:37.293564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:54:37.296148 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:54:37.296120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508d4b1f_e1a7_463c_b176_4de10980b871.slice/crio-a9a5db53f2a65b9be7db9858157def16b0f8c97b9c2a924839e22552de40402f WatchSource:0}: Error finding container a9a5db53f2a65b9be7db9858157def16b0f8c97b9c2a924839e22552de40402f: Status 404 returned error can't find the container with id a9a5db53f2a65b9be7db9858157def16b0f8c97b9c2a924839e22552de40402f
Sep 11 21:54:37.423787 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:37.423761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" event={"ID":"508d4b1f-e1a7-463c-b176-4de10980b871","Type":"ContainerStarted","Data":"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"}
Sep 11 21:54:37.423922 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:37.423796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" event={"ID":"508d4b1f-e1a7-463c-b176-4de10980b871","Type":"ContainerStarted","Data":"a9a5db53f2a65b9be7db9858157def16b0f8c97b9c2a924839e22552de40402f"}
Sep 11 21:54:37.441588 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:37.441540    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" podStartSLOduration=1.4415255359999999 podStartE2EDuration="1.441525536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:37.440398854 +0000 UTC m=+2225.562730773" watchObservedRunningTime="2026-09-11 21:54:37.441525536 +0000 UTC m=+2225.563857447"
Sep 11 21:54:41.198457 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:41.198419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:54:44.104371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:44.104330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:44.104856 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:44.104399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:44.116852 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:44.116829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:44.467994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:44.467914    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:54:47.021048 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:47.021013    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:47.021408 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:47.021061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:47.035393 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:47.035365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:47.479516 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:47.479428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:54:57.525358 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:57.525268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:54:58.488273 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:58.488241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:54:58.864704 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:54:58.864625    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:55:06.592132 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:06.592100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:12.548340 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.548283    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" podUID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" containerName="main" containerID="cri-o://d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec" gracePeriod=45
Sep 11 21:55:12.785421 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.785400    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:55:12.936534 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936534 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936736 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936736 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936736 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75wwj\" (UniqueName: \"kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936736 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir\") pod \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\" (UID: \"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204\") "
Sep 11 21:55:12.936951 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home" (OuterVolumeSpecName: "home") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:12.936951 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936819    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache" (OuterVolumeSpecName: "model-cache") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:12.936951 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.936937    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:12.937099 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.937025    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:12.937099 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.937043    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:12.937099 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.937051    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:12.938991 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.938971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm" (OuterVolumeSpecName: "dshm") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:12.939081 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.939049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj" (OuterVolumeSpecName: "kube-api-access-75wwj") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "kube-api-access-75wwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:12.939153 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:12.939129    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" (UID: "99f9c2e7-4eb2-4450-ae8e-1e01a6b04204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:13.037945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.037923    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wwj\" (UniqueName: \"kubernetes.io/projected/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-kube-api-access-75wwj\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:13.037945 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.037946    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:13.038090 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.037957    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:13.589830 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.589801    2580 generic.go:356] "Generic (PLEG): container finished" podID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" containerID="d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec" exitCode=0
Sep 11 21:55:13.590191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.589844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"
Sep 11 21:55:13.590191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.589852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" event={"ID":"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204","Type":"ContainerDied","Data":"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"}
Sep 11 21:55:13.590191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.589881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk" event={"ID":"99f9c2e7-4eb2-4450-ae8e-1e01a6b04204","Type":"ContainerDied","Data":"e930935405aebfbbe9cff468a977f7c7116cd7c2e703d507a1d036a2d9222d02"}
Sep 11 21:55:13.590191 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.589896    2580 scope.go:117] "RemoveContainer" containerID="d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"
Sep 11 21:55:13.598666 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.598648    2580 scope.go:117] "RemoveContainer" containerID="d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"
Sep 11 21:55:13.598915 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:55:13.598898    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec\": container with ID starting with d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec not found: ID does not exist" containerID="d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"
Sep 11 21:55:13.598978 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.598921    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec"} err="failed to get container status \"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec\": rpc error: code = NotFound desc = could not find container \"d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec\": container with ID starting with d29e7254eca255d1940e0681f543308c1046a885d16150093932a53f43b483ec not found: ID does not exist"
Sep 11 21:55:13.610176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.610151    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:55:13.613696 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.613676    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b4958497-kserve-69f44cb895-kbtqk"]
Sep 11 21:55:13.883168 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:13.883095    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" podUID="508d4b1f-e1a7-463c-b176-4de10980b871" containerName="main" containerID="cri-o://355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0" gracePeriod=45
Sep 11 21:55:14.132676 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.132652    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:55:14.248385 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248403    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx2zp\" (UniqueName: \"kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248562 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir\") pod \"508d4b1f-e1a7-463c-b176-4de10980b871\" (UID: \"508d4b1f-e1a7-463c-b176-4de10980b871\") "
Sep 11 21:55:14.248828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248682    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home" (OuterVolumeSpecName: "home") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:14.248828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248762    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache" (OuterVolumeSpecName: "model-cache") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:14.248828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248788    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-home\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.248828 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.248794    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:14.250639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.250617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:14.250725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.250648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm" (OuterVolumeSpecName: "dshm") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:55:14.250725 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.250661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp" (OuterVolumeSpecName: "kube-api-access-jx2zp") pod "508d4b1f-e1a7-463c-b176-4de10980b871" (UID: "508d4b1f-e1a7-463c-b176-4de10980b871"). InnerVolumeSpecName "kube-api-access-jx2zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:14.349869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.349839    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-tmp-dir\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.349869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.349865    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/508d4b1f-e1a7-463c-b176-4de10980b871-tls-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.349995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.349877    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-dshm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.349995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.349889    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2zp\" (UniqueName: \"kubernetes.io/projected/508d4b1f-e1a7-463c-b176-4de10980b871-kube-api-access-jx2zp\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.349995 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.349900    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/508d4b1f-e1a7-463c-b176-4de10980b871-model-cache\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:14.383867 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.383844    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" path="/var/lib/kubelet/pods/99f9c2e7-4eb2-4450-ae8e-1e01a6b04204/volumes"
Sep 11 21:55:14.595623 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.595556    2580 generic.go:356] "Generic (PLEG): container finished" podID="508d4b1f-e1a7-463c-b176-4de10980b871" containerID="355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0" exitCode=0
Sep 11 21:55:14.595623 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.595609    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"
Sep 11 21:55:14.596074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.595642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" event={"ID":"508d4b1f-e1a7-463c-b176-4de10980b871","Type":"ContainerDied","Data":"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"}
Sep 11 21:55:14.596074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.595690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp" event={"ID":"508d4b1f-e1a7-463c-b176-4de10980b871","Type":"ContainerDied","Data":"a9a5db53f2a65b9be7db9858157def16b0f8c97b9c2a924839e22552de40402f"}
Sep 11 21:55:14.596074 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.595714    2580 scope.go:117] "RemoveContainer" containerID="355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"
Sep 11 21:55:14.603686 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.603661    2580 scope.go:117] "RemoveContainer" containerID="355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"
Sep 11 21:55:14.603951 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:55:14.603929    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0\": container with ID starting with 355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0 not found: ID does not exist" containerID="355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"
Sep 11 21:55:14.603994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.603958    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0"} err="failed to get container status \"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0\": rpc error: code = NotFound desc = could not find container \"355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0\": container with ID starting with 355a0a57308ae8f228680c0023b5fc34a222d5783285e2c2b3553fbdd46f6ef0 not found: ID does not exist"
Sep 11 21:55:14.611639 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.611616    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:55:14.613407 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.613384    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b4958497-kserve-867749f659-hrpsp"]
Sep 11 21:55:14.681141 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.681119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:14.862032 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.861961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:14.862260 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862247    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" containerName="main"
Sep 11 21:55:14.862300 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862261    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" containerName="main"
Sep 11 21:55:14.862300 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862276    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="508d4b1f-e1a7-463c-b176-4de10980b871" containerName="main"
Sep 11 21:55:14.862300 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="508d4b1f-e1a7-463c-b176-4de10980b871" containerName="main"
Sep 11 21:55:14.862406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862339    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="508d4b1f-e1a7-463c-b176-4de10980b871" containerName="main"
Sep 11 21:55:14.862406 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.862349    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f9c2e7-4eb2-4450-ae8e-1e01a6b04204" containerName="main"
Sep 11 21:55:14.866739 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.866720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:14.869427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.869401    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:55:14.869427 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.869424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:55:14.869695 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.869675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:55:14.869765 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.869716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 21:55:14.869821 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.869794    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czlrt\""
Sep 11 21:55:14.875246 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.875216    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"]
Sep 11 21:55:14.882372 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.882349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:14.882476 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.882469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:14.888349 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.888326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"]
Sep 11 21:55:14.955226 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:14.955328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:14.955328 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smz9l\" (UniqueName: \"kubernetes.io/projected/b0f79661-c6c5-4703-8fd2-813fa8e26df6-kube-api-access-smz9l\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:14.955404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-maas-api-tls\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:14.955404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:14.955404 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkpm\" (UniqueName: \"kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:14.955570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-metrics-certs\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:14.955570 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:14.955481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f79661-c6c5-4703-8fd2-813fa8e26df6-openshift-service-ca\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.056196 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.056322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.056322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smz9l\" (UniqueName: \"kubernetes.io/projected/b0f79661-c6c5-4703-8fd2-813fa8e26df6-kube-api-access-smz9l\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.056322 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-maas-api-tls\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.056494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.056494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkpm\" (UniqueName: \"kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.056494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-metrics-certs\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.056494 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.056419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f79661-c6c5-4703-8fd2-813fa8e26df6-openshift-service-ca\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.057277 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.057249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.057277 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.057266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f79661-c6c5-4703-8fd2-813fa8e26df6-openshift-service-ca\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.058847 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.058827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.059011 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.058981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-maas-api-tls\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.059356 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.059336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.059392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.059336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0f79661-c6c5-4703-8fd2-813fa8e26df6-metrics-certs\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.064082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.064057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkpm\" (UniqueName: \"kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm\") pod \"maas-api-ccb8f7bdd-524rw\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.064169 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.064060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smz9l\" (UniqueName: \"kubernetes.io/projected/b0f79661-c6c5-4703-8fd2-813fa8e26df6-kube-api-access-smz9l\") pod \"maas-api-786f4dddd4-whtc9\" (UID: \"b0f79661-c6c5-4703-8fd2-813fa8e26df6\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.183983 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.183920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:15.195017 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.194989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:15.460079 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.459953    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:15.463074 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:55:15.463048    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05279085_e69e_49eb_a7d0_bfae6788ed2e.slice/crio-be096e75477d1ab46b96bf96743f247527cc79bafcdf3dbd387b1575cb025171 WatchSource:0}: Error finding container be096e75477d1ab46b96bf96743f247527cc79bafcdf3dbd387b1575cb025171: Status 404 returned error can't find the container with id be096e75477d1ab46b96bf96743f247527cc79bafcdf3dbd387b1575cb025171
Sep 11 21:55:15.464467 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.464449    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:55:15.476371 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.476349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"]
Sep 11 21:55:15.478289 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:55:15.478265    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f79661_c6c5_4703_8fd2_813fa8e26df6.slice/crio-4a6d603be40b4eab91d880c8cfe88d635cc7d34e77e9bab3728cdc623f168b13 WatchSource:0}: Error finding container 4a6d603be40b4eab91d880c8cfe88d635cc7d34e77e9bab3728cdc623f168b13: Status 404 returned error can't find the container with id 4a6d603be40b4eab91d880c8cfe88d635cc7d34e77e9bab3728cdc623f168b13
Sep 11 21:55:15.602142 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.602114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" event={"ID":"05279085-e69e-49eb-a7d0-bfae6788ed2e","Type":"ContainerStarted","Data":"be096e75477d1ab46b96bf96743f247527cc79bafcdf3dbd387b1575cb025171"}
Sep 11 21:55:15.604101 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:15.604079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9" event={"ID":"b0f79661-c6c5-4703-8fd2-813fa8e26df6","Type":"ContainerStarted","Data":"4a6d603be40b4eab91d880c8cfe88d635cc7d34e77e9bab3728cdc623f168b13"}
Sep 11 21:55:16.383399 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.383366    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="508d4b1f-e1a7-463c-b176-4de10980b871" path="/var/lib/kubelet/pods/508d4b1f-e1a7-463c-b176-4de10980b871/volumes"
Sep 11 21:55:16.610453 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.610417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" event={"ID":"05279085-e69e-49eb-a7d0-bfae6788ed2e","Type":"ContainerStarted","Data":"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"}
Sep 11 21:55:16.610854 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.610549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:16.612203 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.612180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9" event={"ID":"b0f79661-c6c5-4703-8fd2-813fa8e26df6","Type":"ContainerStarted","Data":"e4722aa155b363d7e51b89c028367bb2ef83936c92f1b8a2ea972f6ae1a7a88f"}
Sep 11 21:55:16.612335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.612251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:16.628574 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.628529    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" podStartSLOduration=2.163091858 podStartE2EDuration="2.628515811s" totalImagesPullingTime="465.423953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:16.626731335 +0000 UTC m=+2264.749063247" watchObservedRunningTime="2026-09-11 21:55:16.628515811 +0000 UTC m=+2264.750847722"
Sep 11 21:55:16.643256 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:16.643187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9" podStartSLOduration=2.22381302 podStartE2EDuration="2.643172689s" totalImagesPullingTime="419.359669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:16.642747891 +0000 UTC m=+2264.765079811" watchObservedRunningTime="2026-09-11 21:55:16.643172689 +0000 UTC m=+2264.765504609"
Sep 11 21:55:22.279696 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.279660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:22.621986 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.621909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:22.622403 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.622379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-whtc9"
Sep 11 21:55:22.681982 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.681947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:22.682167 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.682147    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" podUID="05279085-e69e-49eb-a7d0-bfae6788ed2e" containerName="maas-api" containerID="cri-o://1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c" gracePeriod=30
Sep 11 21:55:22.923816 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:22.923794    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:23.022050 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrkpm\" (UniqueName: \"kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm\") pod \"05279085-e69e-49eb-a7d0-bfae6788ed2e\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") "
Sep 11 21:55:23.022224 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca\") pod \"05279085-e69e-49eb-a7d0-bfae6788ed2e\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") "
Sep 11 21:55:23.022224 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls\") pod \"05279085-e69e-49eb-a7d0-bfae6788ed2e\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") "
Sep 11 21:55:23.022224 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs\") pod \"05279085-e69e-49eb-a7d0-bfae6788ed2e\" (UID: \"05279085-e69e-49eb-a7d0-bfae6788ed2e\") "
Sep 11 21:55:23.022460 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05279085-e69e-49eb-a7d0-bfae6788ed2e" (UID: "05279085-e69e-49eb-a7d0-bfae6788ed2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:55:23.022591 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.022575    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05279085-e69e-49eb-a7d0-bfae6788ed2e-openshift-service-ca\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:23.024334 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.024311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm" (OuterVolumeSpecName: "kube-api-access-wrkpm") pod "05279085-e69e-49eb-a7d0-bfae6788ed2e" (UID: "05279085-e69e-49eb-a7d0-bfae6788ed2e"). InnerVolumeSpecName "kube-api-access-wrkpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:55:23.024448 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.024315    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05279085-e69e-49eb-a7d0-bfae6788ed2e" (UID: "05279085-e69e-49eb-a7d0-bfae6788ed2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:23.024448 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.024415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05279085-e69e-49eb-a7d0-bfae6788ed2e" (UID: "05279085-e69e-49eb-a7d0-bfae6788ed2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:55:23.123465 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.123442    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-maas-api-tls\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:23.123465 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.123462    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05279085-e69e-49eb-a7d0-bfae6788ed2e-metrics-certs\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:23.123611 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.123471    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkpm\" (UniqueName: \"kubernetes.io/projected/05279085-e69e-49eb-a7d0-bfae6788ed2e-kube-api-access-wrkpm\") on node \"ip-10-0-135-133.ec2.internal\" DevicePath \"\""
Sep 11 21:55:23.646589 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.646558    2580 generic.go:356] "Generic (PLEG): container finished" podID="05279085-e69e-49eb-a7d0-bfae6788ed2e" containerID="1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c" exitCode=0
Sep 11 21:55:23.646973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.646596    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"
Sep 11 21:55:23.646973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.646631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" event={"ID":"05279085-e69e-49eb-a7d0-bfae6788ed2e","Type":"ContainerDied","Data":"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"}
Sep 11 21:55:23.646973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.646679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw" event={"ID":"05279085-e69e-49eb-a7d0-bfae6788ed2e","Type":"ContainerDied","Data":"be096e75477d1ab46b96bf96743f247527cc79bafcdf3dbd387b1575cb025171"}
Sep 11 21:55:23.646973 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.646700    2580 scope.go:117] "RemoveContainer" containerID="1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"
Sep 11 21:55:23.655419 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.655401    2580 scope.go:117] "RemoveContainer" containerID="1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"
Sep 11 21:55:23.655733 ip-10-0-135-133 kubenswrapper[2580]: E0911 21:55:23.655716    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c\": container with ID starting with 1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c not found: ID does not exist" containerID="1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"
Sep 11 21:55:23.655792 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.655742    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c"} err="failed to get container status \"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c\": rpc error: code = NotFound desc = could not find container \"1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c\": container with ID starting with 1cebbd67736f9956ea29c5ed6f04815a5386b4939a7fc3e5a980e00a178e223c not found: ID does not exist"
Sep 11 21:55:23.667078 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.667059    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:23.670382 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:23.670361    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-524rw"]
Sep 11 21:55:24.383463 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:24.383433    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05279085-e69e-49eb-a7d0-bfae6788ed2e" path="/var/lib/kubelet/pods/05279085-e69e-49eb-a7d0-bfae6788ed2e/volumes"
Sep 11 21:55:28.892444 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:28.892411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:34.733200 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:34.733169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:41.593176 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:41.593146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ss8j8"]
Sep 11 21:55:44.612392 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:44.612365    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f66bb765f-gbm9z_5987d22f-1b14-4211-a1e3-a7f7be363569/authorino/0.log"
Sep 11 21:55:49.978082 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:49.978052    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-klktt_6cad4815-9575-4f2d-bc03-def80dbb4f2d/manager/0.log"
Sep 11 21:55:50.109416 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:50.109390    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m55xd_e1e990a0-3ad7-4963-912f-5263171aed36/manager/0.log"
Sep 11 21:55:50.736312 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:50.736289    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b6c4779-mk7ff_d0073dcb-15f2-4d2f-867d-02d542143d25/copy-manifests/0.log"
Sep 11 21:55:50.750682 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:50.750653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b6c4779-mk7ff_d0073dcb-15f2-4d2f-867d-02d542143d25/manager/0.log"
Sep 11 21:55:51.315709 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:51.315683    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-786f4dddd4-whtc9_b0f79661-c6c5-4703-8fd2-813fa8e26df6/maas-api/0.log"
Sep 11 21:55:51.421609 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:51.421583    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819370-9tk92_d9f4a154-6bc2-4db5-af83-c9205d1accb0/cleanup/0.log"
Sep 11 21:55:52.459576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:52.459549    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f66bb765f-gbm9z_5987d22f-1b14-4211-a1e3-a7f7be363569/authorino/0.log"
Sep 11 21:55:52.577005 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:52.576976    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-k7mtz_33ae7105-b658-41b8-aec1-2c508ff15f2c/manager/0.log"
Sep 11 21:55:52.694370 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:52.694340    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-nqjqh_d4394a32-ee6d-43f5-a240-f9d9d92d145c/manager/0.log"
Sep 11 21:55:52.935668 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:52.935644    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-98jkc_3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062/manager/0.log"
Sep 11 21:55:53.081258 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:53.081230    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ss8j8_ded0665b-ce16-48a1-aa5e-7a5c40b6c645/limitador/0.log"
Sep 11 21:55:53.204306 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:53.204235    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dpfwr_f66f6191-d59c-447a-9b4c-dffba63c19aa/manager/0.log"
Sep 11 21:55:53.596811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:53.596781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2_28dbe602-3995-40a8-aa9e-d17dfa72d340/istio-proxy/0.log"
Sep 11 21:55:53.854359 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:53.854299    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5564d7bb97-8d97v_dbfd34dc-f1d5-47cd-a824-7dcc62450056/kube-auth-proxy/0.log"
Sep 11 21:55:55.278574 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:55.278548    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk99pn_f6822e42-980d-404d-b371-0c288e6fe056/main/0.log"
Sep 11 21:55:55.517272 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:55:55.517245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5gcqt_98453237-c6e0-4e55-88cf-b55181507933/main/0.log"
Sep 11 21:56:03.633443 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:03.633414    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p7n7b_75b7b403-2cec-4496-b4d5-11ac365f0305/global-pull-secret-syncer/0.log"
Sep 11 21:56:03.802698 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:03.802670    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mc442_df250077-d04a-4193-8ab7-758369c6af18/konnectivity-agent/0.log"
Sep 11 21:56:03.823869 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:03.823846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-133.ec2.internal_a7628f9440b1b266be07ea0dcc5e5880/haproxy/0.log"
Sep 11 21:56:09.035237 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.035207    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f66bb765f-gbm9z_5987d22f-1b14-4211-a1e3-a7f7be363569/authorino/0.log"
Sep 11 21:56:09.080343 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.080316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-k7mtz_33ae7105-b658-41b8-aec1-2c508ff15f2c/manager/0.log"
Sep 11 21:56:09.115217 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.115196    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-nqjqh_d4394a32-ee6d-43f5-a240-f9d9d92d145c/manager/0.log"
Sep 11 21:56:09.185111 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.185089    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-98jkc_3f4a8f74-4c04-4c1f-90e5-95d3e9ea6062/manager/0.log"
Sep 11 21:56:09.209152 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.209129    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ss8j8_ded0665b-ce16-48a1-aa5e-7a5c40b6c645/limitador/0.log"
Sep 11 21:56:09.264210 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:09.264190    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dpfwr_f66f6191-d59c-447a-9b4c-dffba63c19aa/manager/0.log"
Sep 11 21:56:11.350670 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:11.350645    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zr42f_735df91f-0758-418c-8361-c7add0082060/node-exporter/0.log"
Sep 11 21:56:11.370086 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:11.370063    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zr42f_735df91f-0758-418c-8361-c7add0082060/kube-rbac-proxy/0.log"
Sep 11 21:56:11.389164 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:11.389145    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zr42f_735df91f-0758-418c-8361-c7add0082060/init-textfile/0.log"
Sep 11 21:56:13.785994 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.785959    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"]
Sep 11 21:56:13.786336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.786268    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05279085-e69e-49eb-a7d0-bfae6788ed2e" containerName="maas-api"
Sep 11 21:56:13.786336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.786279    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05279085-e69e-49eb-a7d0-bfae6788ed2e" containerName="maas-api"
Sep 11 21:56:13.786336 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.786328    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05279085-e69e-49eb-a7d0-bfae6788ed2e" containerName="maas-api"
Sep 11 21:56:13.789119 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.789098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.791576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.791546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xhtjn\"/\"kube-root-ca.crt\""
Sep 11 21:56:13.791576 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.791552    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xhtjn\"/\"openshift-service-ca.crt\""
Sep 11 21:56:13.792492 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.792473    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xhtjn\"/\"default-dockercfg-8csvh\""
Sep 11 21:56:13.798232 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.798211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"]
Sep 11 21:56:13.889750 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.889728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-lib-modules\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.889859 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.889770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-podres\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.889859 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.889787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-proc\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.889859 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.889833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-sys\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.890002 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.889879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptxw\" (UniqueName: \"kubernetes.io/projected/6f398974-ebd2-4ee9-ac35-9ec21602267c-kube-api-access-qptxw\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.990745 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-lib-modules\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.990843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-podres\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.990843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-proc\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.990843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-sys\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.990843 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptxw\" (UniqueName: \"kubernetes.io/projected/6f398974-ebd2-4ee9-ac35-9ec21602267c-kube-api-access-qptxw\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.991044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-lib-modules\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.991044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-podres\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.991044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-proc\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.991044 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.990930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f398974-ebd2-4ee9-ac35-9ec21602267c-sys\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:13.998474 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:13.998452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptxw\" (UniqueName: \"kubernetes.io/projected/6f398974-ebd2-4ee9-ac35-9ec21602267c-kube-api-access-qptxw\") pod \"perf-node-gather-daemonset-zbpx6\" (UID: \"6f398974-ebd2-4ee9-ac35-9ec21602267c\") " pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:14.099279 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.099228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:14.366193 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.366105    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"]
Sep 11 21:56:14.368352 ip-10-0-135-133 kubenswrapper[2580]: W0911 21:56:14.368325    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f398974_ebd2_4ee9_ac35_9ec21602267c.slice/crio-e7a94f3695fb91def6343ca84be2f584a0fa016b5038c0e2d97241a506f81f0e WatchSource:0}: Error finding container e7a94f3695fb91def6343ca84be2f584a0fa016b5038c0e2d97241a506f81f0e: Status 404 returned error can't find the container with id e7a94f3695fb91def6343ca84be2f584a0fa016b5038c0e2d97241a506f81f0e
Sep 11 21:56:14.875318 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.875290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6" event={"ID":"6f398974-ebd2-4ee9-ac35-9ec21602267c","Type":"ContainerStarted","Data":"c15c7d1d032c472dd80f6d1b1fd6ad2672d87acde42e73d5b03aaa5b2dd51c3d"}
Sep 11 21:56:14.875678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.875338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6" event={"ID":"6f398974-ebd2-4ee9-ac35-9ec21602267c","Type":"ContainerStarted","Data":"e7a94f3695fb91def6343ca84be2f584a0fa016b5038c0e2d97241a506f81f0e"}
Sep 11 21:56:14.875678 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.875384    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:14.890801 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:14.890761    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6" podStartSLOduration=1.890745141 podStartE2EDuration="1.890745141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:14.889551326 +0000 UTC m=+2323.011883247" watchObservedRunningTime="2026-09-11 21:56:14.890745141 +0000 UTC m=+2323.013077062"
Sep 11 21:56:16.108400 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:16.108370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ht7j2_b4da237f-c80d-485a-a623-53480965fe1d/dns/0.log"
Sep 11 21:56:16.127661 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:16.127637    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ht7j2_b4da237f-c80d-485a-a623-53480965fe1d/kube-rbac-proxy/0.log"
Sep 11 21:56:16.195703 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:16.195684    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mmbt_9bf02f91-8277-4937-a4cf-eda0b2e0c0d8/dns-node-resolver/0.log"
Sep 11 21:56:16.763864 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:16.763835    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f578bb4fd-4dphg_a388ba12-8b45-4705-a47b-b718205da999/registry/0.log"
Sep 11 21:56:16.808357 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:16.808333    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j4mfs_c2a51e6e-8e64-4ae8-a1c4-9207f0322e19/node-ca/0.log"
Sep 11 21:56:17.759811 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:17.759786    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2_28dbe602-3995-40a8-aa9e-d17dfa72d340/istio-proxy/0.log"
Sep 11 21:56:18.612436 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:18.612405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5564d7bb97-8d97v_dbfd34dc-f1d5-47cd-a824-7dcc62450056/kube-auth-proxy/0.log"
Sep 11 21:56:19.776098 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:19.776071    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pzf2m_38eddb37-c2bc-4554-b33b-f38af8bac54f/serve-healthcheck-canary/0.log"
Sep 11 21:56:20.350281 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:20.350246    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q8s9c_f55297f1-2172-4f28-affa-157d70a431c1/kube-rbac-proxy/0.log"
Sep 11 21:56:20.368988 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:20.368964    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q8s9c_f55297f1-2172-4f28-affa-157d70a431c1/exporter/0.log"
Sep 11 21:56:20.387846 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:20.387824    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q8s9c_f55297f1-2172-4f28-affa-157d70a431c1/extractor/0.log"
Sep 11 21:56:20.888625 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:20.888599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xhtjn/perf-node-gather-daemonset-zbpx6"
Sep 11 21:56:22.761204 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:22.761182    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-klktt_6cad4815-9575-4f2d-bc03-def80dbb4f2d/manager/0.log"
Sep 11 21:56:23.025342 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:23.025318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m55xd_e1e990a0-3ad7-4963-912f-5263171aed36/manager/0.log"
Sep 11 21:56:23.380876 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:23.380804    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b6c4779-mk7ff_d0073dcb-15f2-4d2f-867d-02d542143d25/manager/0.log"
Sep 11 21:56:23.395810 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:23.395781    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b6c4779-mk7ff_d0073dcb-15f2-4d2f-867d-02d542143d25/copy-manifests/0.log"
Sep 11 21:56:24.252954 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:24.252918    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2_28dbe602-3995-40a8-aa9e-d17dfa72d340/istio-proxy/0.log"
Sep 11 21:56:24.735286 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:24.735245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tgcgm_e8f7f3c5-42e6-4523-96f6-ce5ee9a4a065/openshift-lws-operator/0.log"
Sep 11 21:56:25.098335 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:25.098309    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5564d7bb97-8d97v_dbfd34dc-f1d5-47cd-a824-7dcc62450056/kube-auth-proxy/0.log"
Sep 11 21:56:26.672166 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:26.672134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cbb5b2_28dbe602-3995-40a8-aa9e-d17dfa72d340/istio-proxy/0.log"
Sep 11 21:56:27.572637 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:27.572599    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5564d7bb97-8d97v_dbfd34dc-f1d5-47cd-a824-7dcc62450056/kube-auth-proxy/0.log"
Sep 11 21:56:31.313043 ip-10-0-135-133 kubenswrapper[2580]: I0911 21:56:31.313010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6hbc5_a5747e3b-6c03-4944-a051-520a9910f76e/kube-multus/0.log"