Jul 30 03:45:05.421164 ip-10-0-138-44 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 03:45:05.730555 ip-10-0-138-44 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:45:05.730555 ip-10-0-138-44 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 03:45:05.730555 ip-10-0-138-44 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:45:05.730555 ip-10-0-138-44 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 03:45:05.730555 ip-10-0-138-44 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 03:45:05.732562 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.732487    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 03:45:05.736121 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736107    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.736121 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736120    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736124    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736127    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736129    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736133    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736135    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736138    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736140    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736143    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736145    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736148    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736151    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736154    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736157    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736160    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736163    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736166    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736168    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736174    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736176    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.736185 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736179    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736182    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736184    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736186    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736190    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736192    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736195    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736198    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736201    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736203    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736207    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736211    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736214    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736217    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736220    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736223    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736226    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736228    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736230    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.736637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736233    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736235    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736238    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736240    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736242    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736245    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736247    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736250    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736252    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736254    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736257    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736260    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736262    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736265    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736267    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736273    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736276    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736279    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736286    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.737094 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736289    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736292    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736295    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736298    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736301    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736304    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736306    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736309    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736311    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736314    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736316    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736319    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736322    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736324    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736327    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736329    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736333    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736335    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736338    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736340    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.737534 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736343    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736345    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736347    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736350    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736353    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736355    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736358    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736360    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736363    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736366    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736368    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736371    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736373    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736376    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736378    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736380    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736383    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736385    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736388    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.738013 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.736390    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.739597 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739585    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.739597 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739597    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739600    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739604    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739608    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739611    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739613    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739616    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739618    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739621    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739623    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739626    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739629    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739631    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739634    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739637    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739639    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739642    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739644    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739646    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739649    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.739656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739652    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739654    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739656    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739661    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739664    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739667    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739670    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739673    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739676    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739678    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739682    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739685    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739688    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739691    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739693    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739697    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739701    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739704    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739707    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.740137 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739709    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739712    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739714    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739717    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739719    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739722    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739724    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739726    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739729    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739731    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739733    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739736    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739738    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739741    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739743    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739746    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739749    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739752    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739754    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739757    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.740585 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739759    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739762    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739764    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739767    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739769    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739772    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739774    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739777    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739779    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739782    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739784    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739787    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739789    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739792    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739795    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739797    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739799    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739802    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739804    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739807    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.741051 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739809    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739812    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739814    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739817    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739819    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739822    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739825    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739828    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739831    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739835    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739838    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739841    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739844    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739846    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739848    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739851    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739854    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739856    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.739859    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.741519 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740199    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740203    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740207    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740210    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740212    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740215    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740218    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740220    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740223    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740225    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740228    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740230    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740235    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740239    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740242    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740244    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740247    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740250    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740252    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.741978 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740255    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740258    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740260    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740263    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740265    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740284    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740289    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740292    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740295    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740298    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740301    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740304    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740307    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740309    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740314    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740317    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740321    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740324    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740327    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.742422 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740330    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740332    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740334    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740337    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740339    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740341    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740345    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740347    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740350    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740352    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740355    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740357    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740360    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740362    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740365    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740368    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740370    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740373    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740375    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740377    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.742866 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740380    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740383    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740385    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740388    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740390    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740393    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740395    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740397    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740400    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740402    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740405    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740407    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740410    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740412    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740414    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740417    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740419    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740423    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740426    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740429    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.743337 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740431    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740434    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740436    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740439    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740441    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740443    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740446    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740448    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740451    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740453    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740455    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740458    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740460    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740463    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740465    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740467    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740470    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740472    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740475    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740477    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.743832 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.740480    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740544    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740551    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740558    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740564    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740570    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740575    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740581    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740586    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740591    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740599    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740602    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740606    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740609    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740611    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740614    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740617    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740620    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740623    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740640    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740645    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740649    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740652    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740656    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 03:45:05.744310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740660    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740662    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740666    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740669    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740672    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740675    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740677    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740680    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740683    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740687    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740690    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740693    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740696    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740699    2562 flags.go:64] FLAG: --enable-server="true"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740702    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740705    2562 flags.go:64] FLAG: --event-burst="100"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740708    2562 flags.go:64] FLAG: --event-qps="50"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740711    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740717    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740719    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740723    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740726    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740729    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740732    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740734    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 03:45:05.744853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740737    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740740    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740743    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740746    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740749    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740751    2562 flags.go:64] FLAG: --feature-gates=""
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740755    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740758    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740761    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740764    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740767    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740770    2562 flags.go:64] FLAG: --help="false"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740773    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-138-44.ec2.internal"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740775    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740779    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740781    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740784    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740788    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740790    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740793    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740799    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740802    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740805    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740808    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 03:45:05.745479 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740810    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740816    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740818    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740821    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740824    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740827    2562 flags.go:64] FLAG: --lock-file=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740829    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740832    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740835    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740845    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740847    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740850    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740853    2562 flags.go:64] FLAG: --logging-format="text"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740856    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740859    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740861    2562 flags.go:64] FLAG: --manifest-url=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740864    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740868    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740871    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740874    2562 flags.go:64] FLAG: --max-pods="110"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740877    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740880    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740883    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740885    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740888    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 03:45:05.746161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740903    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740906    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740913    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740916    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740920    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740923    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740926    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740930    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740935    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740939    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740941    2562 flags.go:64] FLAG: --port="10250"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740944    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740947    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0210fd94e65970328"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740951    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740954    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740957    2562 flags.go:64] FLAG: --register-node="true"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740959    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740963    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740965    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740969    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740971    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740975    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740978    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740981    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740983    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 03:45:05.747073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740986    2562 flags.go:64] FLAG: --runonce="false"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740989    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740992    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740994    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.740997    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741000    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741003    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741006    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741009    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741012    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741015    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741018    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741021    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741024    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741027    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741031    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741040    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741042    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741045    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741050    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741053    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741056    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741059    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741062    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741065    2562 flags.go:64] FLAG: --v="2"
Jul 30 03:45:05.748074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741068    2562 flags.go:64] FLAG: --version="false"
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741072    2562 flags.go:64] FLAG: --vmodule=""
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741075    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741078    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741192    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741195    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741199    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741202    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741204    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741207    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741210    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741212    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741215    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741217    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741220    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741222    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741225    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741228    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741230    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741233    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.749003 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741236    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741238    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741243    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741248    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741250    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741253    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741255    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741258    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741260    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741263    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741265    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741268    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741270    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741273    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741275    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741278    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741282    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741285    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741288    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.749469 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741291    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741294    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741297    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741300    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741303    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741305    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741308    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741310    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741313    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741315    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741318    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741320    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741323    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741325    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741329    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741333    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741337    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741339    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741342    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741344    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.749968 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741347    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741350    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741352    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741355    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741357    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741360    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741362    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741365    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741367    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741370    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741372    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741375    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741378    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741380    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741382    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741385    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741387    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741390    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741392    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.750438 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741395    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741397    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741400    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741402    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741405    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741407    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741410    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741412    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741417    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741421    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741424    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741426    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741429    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741431    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741434    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741436    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741439    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741441    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741443    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741446    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.750875 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741450    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741453    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741456    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741458    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.741461    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.741968    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.749709    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.749723    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.750373    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.750384    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750439    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750445    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750448    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750451    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.751356 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750454    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750456    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750459    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750461    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750464    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750466    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750469    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750471    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750474    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750476    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750479    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750481    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750484    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750486    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750489    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750491    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750494    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750496    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750499    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750501    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.751688 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750504    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750506    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750509    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750512    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750516    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750520    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750524    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750527    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750529    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750532    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750534    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750537    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750539    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750542    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750545    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750549    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750552    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750554    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750557    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.752221 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750559    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750562    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750564    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750568    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750570    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750573    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750576    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750578    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750581    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750583    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750586    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750589    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750592    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750595    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750597    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750600    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750602    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750605    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750608    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750610    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.752669 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750613    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750615    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750618    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750620    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750622    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750625    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750628    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750631    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750634    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750638    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750641    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750643    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750646    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750649    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750651    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750654    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750657    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750660    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750662    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.753180 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750664    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750667    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750670    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750672    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750675    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750686    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750688    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750691    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750693    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750696    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750698    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750701    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750704    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750706    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750709    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750711    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.753614 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750714    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.750719    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750826    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750832    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750835    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750838    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750840    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750843    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750846    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750848    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750851    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750853    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750855    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750858    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750861    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 03:45:05.754017 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750863    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750866    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750868    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750871    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750874    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750877    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750879    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750882    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750884    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750887    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750900    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750903    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750906    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750908    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750911    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750914    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750918    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750922    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750924    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 03:45:05.754361 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750927    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750929    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750932    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750934    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750937    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750939    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750942    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750944    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750948    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750951    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750954    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750957    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750959    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750962    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750965    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750968    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750970    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750973    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750975    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750978    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 03:45:05.754801 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750980    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750983    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750985    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750988    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750990    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750993    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750995    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.750998    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751000    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751002    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751006    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751009    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751012    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751014    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751017    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751019    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751022    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751024    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751026    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 03:45:05.755272 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751029    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751031    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751034    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751036    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751038    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751041    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751043    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751046    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751049    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751051    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751054    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751056    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751058    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751061    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751063    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751066    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751068    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751071    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751073    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751076    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 03:45:05.755702 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751078    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751080    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751083    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751085    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751088    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751091    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751093    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:05.751095    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.751100    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.751186    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.752943    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.753923    2562 server.go:1019] "Starting client certificate rotation"
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.754021    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 03:45:05.756182 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.754058    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 03:45:05.756489 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.755803    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 03:45:05.758260 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.758246    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 03:45:05.766726 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.766707    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 30 03:45:05.770431 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.770416    2562 log.go:25] "Validated CRI v1 image API"
Jul 30 03:45:05.771306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.771292    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 03:45:05.775206 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.775189    2562 fs.go:135] Filesystem UUIDs: map[0673726b-d59e-4a24-b00c-d4c7080b31c7:/dev/nvme0n1p4 141baf0c-e56f-489b-be01-1180eca26c67:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 30 03:45:05.775267 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.775205    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 03:45:05.781798 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.781665    2562 manager.go:216] Machine: {Timestamp:2026-07-30 03:45:05.780867825 +0000 UTC m=+0.265431673 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197363 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec209d316d86edc7e91e016ac3aeb9fb SystemUUID:ec209d31-6d86-edc7-e91e-016ac3aeb9fb BootID:c005089b-17c6-4e2c-ada6-1a6f38ef5a4d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:98:83:74:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:98:83:74:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:cf:04:15:b7:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 03:45:05.781798 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.781787    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 03:45:05.781919 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.781866    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 03:45:05.783502 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.783484    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 03:45:05.784116 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.784093    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 03:45:05.784263 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.784123    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 03:45:05.784776 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.784764    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 03:45:05.784834 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.784782    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 03:45:05.784834 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.784792    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 03:45:05.785916 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.785902    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 03:45:05.803450 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.803431    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v687c"
Jul 30 03:45:05.811215 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.811198    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v687c"
Jul 30 03:45:05.910984 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.910965    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 03:45:05.915414 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915399    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 03:45:05.915548 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915538    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 03:45:05.915610 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915575    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 03:45:05.915610 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915588    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 03:45:05.915610 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915602    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 03:45:05.915720 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915615    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 30 03:45:05.915720 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915627    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 03:45:05.915975 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915958    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 03:45:05.916062 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.915982    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 03:45:05.917869 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.917852    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 03:45:05.919111 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919096    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 03:45:05.919172 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919126    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 03:45:05.919678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919660    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 03:45:05.919678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919679    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919686    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919691    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919699    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919704    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919715    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919722    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919729    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919734    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919742    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 03:45:05.919782 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.919747    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 03:45:05.920587 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.920577    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 03:45:05.920587 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.920587    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 03:45:05.923577 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.923563    2562 server.go:1268] "Started kubelet"
Jul 30 03:45:05.924053 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.923990    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 03:45:05.924137 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.924071    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 03:45:05.924197 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.924168    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 03:45:05.924366 ip-10-0-138-44 systemd[1]: Started Kubernetes Kubelet.
Jul 30 03:45:05.925513 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.925505    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 03:45:05.926011 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.925992    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 03:45:05.928014 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.927993    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:05.932676 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.932656    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 03:45:05.932754 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.932700    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 03:45:05.932877 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.932783    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 03:45:05.932951 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.932933    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-44.ec2.internal\" not found"
Jul 30 03:45:05.933863 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.933843    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 03:45:05.933863 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.933863    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 03:45:05.933996 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.933922    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 03:45:05.934054 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934023    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 03:45:05.934054 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934031    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 03:45:05.934371 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934353    2562 factory.go:55] Registering systemd factory
Jul 30 03:45:05.934443 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934418    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 30 03:45:05.934495 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934435    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:05.934957 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934939    2562 factory.go:153] Registering CRI-O factory
Jul 30 03:45:05.935042 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.934960    2562 factory.go:223] Registration of the crio container factory successfully
Jul 30 03:45:05.935042 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.935018    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 03:45:05.935042 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.935041    2562 factory.go:103] Registering Raw factory
Jul 30 03:45:05.935187 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.935069    2562 manager.go:1196] Started watching for new ooms in manager
Jul 30 03:45:05.935187 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.935109    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:05.935441 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.935427    2562 manager.go:349] Starting recovery of all containers
Jul 30 03:45:05.936907 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.936880    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-44.ec2.internal" not found
Jul 30 03:45:05.939152 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.939128    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-44.ec2.internal\" not found" node="ip-10-0-138-44.ec2.internal"
Jul 30 03:45:05.944599 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.944578    2562 manager.go:354] Recovery completed
Jul 30 03:45:05.946071 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.946047    2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 30 03:45:05.949251 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.949238    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:45:05.950925 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.950886    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:45:05.951023 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.950929    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:45:05.951023 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.950943    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:45:05.951484 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.951472    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 03:45:05.951484 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.951483    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 03:45:05.951574 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.951490    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:45:05.951574 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.951499    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 03:45:05.952127 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.952113    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:45:05.952196 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.952137    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:45:05.952196 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.952149    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:45:05.953003 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.952992    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-44.ec2.internal" not found
Jul 30 03:45:05.953637 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.953625    2562 policy_none.go:49] "None policy: Start"
Jul 30 03:45:05.953669 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.953641    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 03:45:05.953669 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.953651    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 03:45:05.955565 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.955554    2562 policy_none.go:47] "Start"
Jul 30 03:45:05.992525 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.992387    2562 manager.go:337] "Starting Device Plugin manager"
Jul 30 03:45:05.992604 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.992526    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 03:45:05.992604 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.992540    2562 server.go:87] "Starting device plugin registration server"
Jul 30 03:45:05.992731 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.992719    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 03:45:05.992759 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.992733    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 03:45:05.993439 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.992827    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 03:45:05.993439 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.993185    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 03:45:05.993439 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:05.993194    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 03:45:05.993819 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.993798    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 03:45:05.993938 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:05.993855    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-44.ec2.internal\" not found"
Jul 30 03:45:06.008918 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.008889    2562 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-44.ec2.internal" not found
Jul 30 03:45:06.093100 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.093080    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 03:45:06.095026 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.093776    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 03:45:06.095026 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.093798    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 03:45:06.095026 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.093807    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeHasSufficientPID"
Jul 30 03:45:06.095026 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.093826    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.102287 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.102273    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.109663 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.109642    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 03:45:06.110832 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.110815    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 03:45:06.110937 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.110862    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 03:45:06.110937 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.110887    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 03:45:06.111048 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:06.110939    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 03:45:06.112823 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.112802    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:06.211773 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.211748    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"]
Jul 30 03:45:06.214111 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.214095    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.216041 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.216024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.235205 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.235185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28ee2cd84792d9409e5acf4090158792-config\") pod \"kube-apiserver-proxy-ip-10-0-138-44.ec2.internal\" (UID: \"28ee2cd84792d9409e5acf4090158792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.235275 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.235208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.235275 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.235225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.241345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.241331    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.245267 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.245253    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.250067 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.250055    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 03:45:06.255027 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.255015    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 03:45:06.336187 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28ee2cd84792d9409e5acf4090158792-config\") pod \"kube-apiserver-proxy-ip-10-0-138-44.ec2.internal\" (UID: \"28ee2cd84792d9409e5acf4090158792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.336264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.336264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.336264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.336364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28ee2cd84792d9409e5acf4090158792-config\") pod \"kube-apiserver-proxy-ip-10-0-138-44.ec2.internal\" (UID: \"28ee2cd84792d9409e5acf4090158792\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:06.336364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:06.336278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1f25fc1428d45d3789fd496fdfa89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal\" (UID: \"1b1f25fc1428d45d3789fd496fdfa89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:08.220160 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.220123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:08.225960 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.225947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal"
Jul 30 03:45:08.421799 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.421777    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 03:45:08.422206 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.421892    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:45:08.422206 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.421923    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:45:08.422206 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.421943    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 03:45:08.583590 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.583566    2562 apiserver.go:52] "Watching apiserver"
Jul 30 03:45:08.583680 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.583551    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 03:40:07 +0000 UTC" deadline="2028-04-20 05:55:03.806785872 +0000 UTC"
Jul 30 03:45:08.583680 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.583663    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15122h9m55.223126796s"
Jul 30 03:45:08.591264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.591247    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 03:45:08.592268 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.592250    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-v6tcn","openshift-cluster-node-tuning-operator/tuned-xv4fs","openshift-image-registry/node-ca-cbzzj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal","openshift-multus/network-metrics-daemon-svjms","openshift-network-operator/iptables-alerter-9h697","openshift-ovn-kubernetes/ovnkube-node-sq668","kube-system/global-pull-secret-syncer-wfpd9","kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn","openshift-dns/node-resolver-zrzfr","openshift-multus/multus-6nnwv","openshift-multus/multus-additional-cni-plugins-lgjs4","openshift-network-diagnostics/network-check-target-s5n46"]
Jul 30 03:45:08.594605 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.594590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.596763 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.596746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.597003 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.596982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hvhkn\""
Jul 30 03:45:08.597089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.596991    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 03:45:08.597089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.597022    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 03:45:08.598780 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.598764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.598970 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.598947    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.599043 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.599013    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87lxp\""
Jul 30 03:45:08.599231 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.599216    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.600804 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.600791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.600878 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.600854    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:08.601278 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.601251    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 03:45:08.601497 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.601480    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.601585 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.601544    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rrb25\""
Jul 30 03:45:08.601585 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.601552    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.601682 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.601627    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 03:45:08.604248 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.604232    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.606166 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606152    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.606225 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606178    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.606225 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606190    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 03:45:08.606406 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606392    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4p6c\""
Jul 30 03:45:08.606457 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.606509 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-var-lib-kubelet\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606557 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-host\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606588 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.606623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4f5\" (UniqueName: \"kubernetes.io/projected/b7d87d97-1e85-402c-a297-1d98cbaa8dba-kube-api-access-9m4f5\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.606623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysconfig\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606714 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-sys\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606714 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7d87d97-1e85-402c-a297-1d98cbaa8dba-host-slash\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.606785 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3706bd53-387b-41be-8596-e51f050f920e-agent-certs\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.606785 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606785 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-systemd\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-run\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-tmp\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lq9\" (UniqueName: \"kubernetes.io/projected/23f6c619-1847-44a2-a21e-a20912d358cd-kube-api-access-x6lq9\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28a9969-37d4-4242-a701-75478e599497-host\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-modprobe-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.606894 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-conf\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-lib-modules\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-etc-tuned\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.606976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28a9969-37d4-4242-a701-75478e599497-serviceca\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.607007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqdq\" (UniqueName: \"kubernetes.io/projected/d28a9969-37d4-4242-a701-75478e599497-kube-api-access-lsqdq\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.607022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhg9\" (UniqueName: \"kubernetes.io/projected/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-kube-api-access-vxhg9\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.607042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7d87d97-1e85-402c-a297-1d98cbaa8dba-iptables-alerter-script\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.607057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3706bd53-387b-41be-8596-e51f050f920e-konnectivity-ca\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.607089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.607078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-kubernetes\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.608500 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.608483    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 03:45:08.608699 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.608686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.608760 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.608740    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:08.608889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.608874    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.608995 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.608978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 03:45:08.609044 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.608999    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.609171 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.609158    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 03:45:08.610163 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.610145    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7gjf\""
Jul 30 03:45:08.610255 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.610170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 03:45:08.610967 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.610947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.613135 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613120    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.613217 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-drcks\""
Jul 30 03:45:08.613276 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613222    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 03:45:08.613276 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613230    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.613354 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613199    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.613826 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.613813    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 03:45:08.614922 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.614911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wcz7j\""
Jul 30 03:45:08.615135 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.615123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.615381 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.615367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.615440 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.615428    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.617623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617607    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 03:45:08.617721 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 03:45:08.617788 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 03:45:08.617844 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fr24d\""
Jul 30 03:45:08.617844 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.617934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.617834    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 03:45:08.620050 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.620034    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 03:45:08.620104 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.620086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:08.620159 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.620146    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dn44q\""
Jul 30 03:45:08.620159 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.620147    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:08.620351 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.620338    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 03:45:08.633578 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.633561    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lpqzq"
Jul 30 03:45:08.643242 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.643226    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lpqzq"
Jul 30 03:45:08.703373 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.703358    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 03:45:08.707873 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-etc-kubernetes\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.707962 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysconfig\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.707962 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.707962 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pm7\" (UniqueName: \"kubernetes.io/projected/efb3b2a5-0f80-4320-b493-a18d2917a8f3-kube-api-access-h9pm7\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.708080 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-systemd-units\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708080 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.707999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-ovn\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708080 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysconfig\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-conf-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-systemd\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-tmp\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-registration-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-systemd\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-netns\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-conf\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-lib-modules\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-kubelet-config\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-os-release\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708344    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-conf\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-system-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-lib-modules\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3706bd53-387b-41be-8596-e51f050f920e-konnectivity-ca\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-kubernetes\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708466    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-host\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-kubernetes\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4f5\" (UniqueName: \"kubernetes.io/projected/b7d87d97-1e85-402c-a297-1d98cbaa8dba-kube-api-access-9m4f5\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-kubelet\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-host\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-var-lib-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-log-socket\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35991ee7-db6e-4812-94fc-3d81199178b6-hosts-file\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-script-lib\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs59j\" (UniqueName: \"kubernetes.io/projected/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-kube-api-access-xs59j\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-daemon-config\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7d87d97-1e85-402c-a297-1d98cbaa8dba-host-slash\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-run\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lq9\" (UniqueName: \"kubernetes.io/projected/23f6c619-1847-44a2-a21e-a20912d358cd-kube-api-access-x6lq9\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.708946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpct\" (UniqueName: \"kubernetes.io/projected/35991ee7-db6e-4812-94fc-3d81199178b6-kube-api-access-ndpct\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7d87d97-1e85-402c-a297-1d98cbaa8dba-host-slash\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-run\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3706bd53-387b-41be-8596-e51f050f920e-konnectivity-ca\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-sysctl-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.708997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-etc-tuned\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqdq\" (UniqueName: \"kubernetes.io/projected/d28a9969-37d4-4242-a701-75478e599497-kube-api-access-lsqdq\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhg9\" (UniqueName: \"kubernetes.io/projected/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-kube-api-access-vxhg9\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-device-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-cni-binary-copy\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-var-lib-kubelet\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-sys-fs\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-system-cni-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-os-release\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.709645 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-multus\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-var-lib-kubelet\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-sys\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-bin\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-config\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-sys\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-dbus\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cnibin\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-slash\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-etc-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-node-log\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-k8s-cni-cncf-io\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-cnibin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-socket-dir-parent\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709653    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-socket-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.710264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-hostroot\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-multus-certs\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.709711    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35991ee7-db6e-4812-94fc-3d81199178b6-tmp-dir\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l5qn\" (UniqueName: \"kubernetes.io/projected/49a91781-5cb3-40c3-acba-91c9b1e807c4-kube-api-access-6l5qn\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.709812    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:09.209765147 +0000 UTC m=+2.026701605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28a9969-37d4-4242-a701-75478e599497-serviceca\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3706bd53-387b-41be-8596-e51f050f920e-agent-certs\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28a9969-37d4-4242-a701-75478e599497-host\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.709977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-etc-selinux\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28a9969-37d4-4242-a701-75478e599497-host\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-netd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-bin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-kubelet\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-modprobe-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7d87d97-1e85-402c-a297-1d98cbaa8dba-iptables-alerter-script\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.710847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d28a9969-37d4-4242-a701-75478e599497-serviceca\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-env-overrides\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/23f6c619-1847-44a2-a21e-a20912d358cd-etc-modprobe-d\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrrb\" (UniqueName: \"kubernetes.io/projected/4999b8fa-fb60-4155-97ac-856262bd6ea3-kube-api-access-5jrrb\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-netns\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-systemd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.711356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.710634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7d87d97-1e85-402c-a297-1d98cbaa8dba-iptables-alerter-script\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.711970 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.711952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-etc-tuned\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.712043 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.711959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23f6c619-1847-44a2-a21e-a20912d358cd-tmp\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.712487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.712472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3706bd53-387b-41be-8596-e51f050f920e-agent-certs\") pod \"konnectivity-agent-v6tcn\" (UID: \"3706bd53-387b-41be-8596-e51f050f920e\") " pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.715508 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.715489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4f5\" (UniqueName: \"kubernetes.io/projected/b7d87d97-1e85-402c-a297-1d98cbaa8dba-kube-api-access-9m4f5\") pod \"iptables-alerter-9h697\" (UID: \"b7d87d97-1e85-402c-a297-1d98cbaa8dba\") " pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.715720 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.715706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqdq\" (UniqueName: \"kubernetes.io/projected/d28a9969-37d4-4242-a701-75478e599497-kube-api-access-lsqdq\") pod \"node-ca-cbzzj\" (UID: \"d28a9969-37d4-4242-a701-75478e599497\") " pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.715841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.715828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lq9\" (UniqueName: \"kubernetes.io/projected/23f6c619-1847-44a2-a21e-a20912d358cd-kube-api-access-x6lq9\") pod \"tuned-xv4fs\" (UID: \"23f6c619-1847-44a2-a21e-a20912d358cd\") " pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.716370 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.716354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhg9\" (UniqueName: \"kubernetes.io/projected/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-kube-api-access-vxhg9\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:08.810681 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-netd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-bin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-kubelet\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-bin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-netd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-env-overrides\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-kubelet\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.810852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrrb\" (UniqueName: \"kubernetes.io/projected/4999b8fa-fb60-4155-97ac-856262bd6ea3-kube-api-access-5jrrb\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-netns\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.810998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-systemd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-etc-kubernetes\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-netns\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pm7\" (UniqueName: \"kubernetes.io/projected/efb3b2a5-0f80-4320-b493-a18d2917a8f3-kube-api-access-h9pm7\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-systemd-units\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-etc-kubernetes\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-systemd\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-systemd-units\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-ovn\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-conf-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.811240    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-registration-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-netns\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811306 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-ovn\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.811298    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:09.311278499 +0000 UTC m=+2.128214974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-conf-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-netns\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-registration-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-kubelet-config\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-os-release\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-env-overrides\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-kubelet-config\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-system-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-os-release\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-kubelet\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-system-cni-dir\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-kubelet\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.811934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/efb3b2a5-0f80-4320-b493-a18d2917a8f3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.811987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-var-lib-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-var-lib-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-log-socket\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35991ee7-db6e-4812-94fc-3d81199178b6-hosts-file\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-log-socket\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-script-lib\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812169    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35991ee7-db6e-4812-94fc-3d81199178b6-hosts-file\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs59j\" (UniqueName: \"kubernetes.io/projected/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-kube-api-access-xs59j\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-daemon-config\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpct\" (UniqueName: \"kubernetes.io/projected/35991ee7-db6e-4812-94fc-3d81199178b6-kube-api-access-ndpct\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-device-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.812692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-cni-binary-copy\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-sys-fs\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-system-cni-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-run-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-system-cni-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-device-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-sys-fs\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-os-release\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-multus\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-bin\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-os-release\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-config\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-var-lib-cni-multus\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-dbus\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.813348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cnibin\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-slash\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-etc-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-node-log\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-k8s-cni-cncf-io\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-script-lib\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812791    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-daemon-config\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-cnibin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-socket-dir-parent\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-cnibin\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49a91781-5cb3-40c3-acba-91c9b1e807c4-cni-binary-copy\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-cni-bin\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-host-slash\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-socket-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efb3b2a5-0f80-4320-b493-a18d2917a8f3-cnibin\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-dbus\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-hostroot\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.813853 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-k8s-cni-cncf-io\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-multus-certs\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-etc-openvswitch\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-multus-socket-dir-parent\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-hostroot\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-socket-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35991ee7-db6e-4812-94fc-3d81199178b6-tmp-dir\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/49a91781-5cb3-40c3-acba-91c9b1e807c4-host-run-multus-certs\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-node-log\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.812993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l5qn\" (UniqueName: \"kubernetes.io/projected/49a91781-5cb3-40c3-acba-91c9b1e807c4-kube-api-access-6l5qn\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-etc-selinux\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4999b8fa-fb60-4155-97ac-856262bd6ea3-etc-selinux\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35991ee7-db6e-4812-94fc-3d81199178b6-tmp-dir\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovn-node-metrics-cert\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.814327 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.813617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-ovnkube-config\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.817664 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.817645    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:08.817664 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.817665    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:08.817808 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.817677    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:08.817808 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:08.817726    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:09.317714594 +0000 UTC m=+2.134651061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:08.819345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.819319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrrb\" (UniqueName: \"kubernetes.io/projected/4999b8fa-fb60-4155-97ac-856262bd6ea3-kube-api-access-5jrrb\") pod \"aws-ebs-csi-driver-node-sk9gn\" (UID: \"4999b8fa-fb60-4155-97ac-856262bd6ea3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.821137 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.819987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpct\" (UniqueName: \"kubernetes.io/projected/35991ee7-db6e-4812-94fc-3d81199178b6-kube-api-access-ndpct\") pod \"node-resolver-zrzfr\" (UID: \"35991ee7-db6e-4812-94fc-3d81199178b6\") " pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.821137 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.820592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l5qn\" (UniqueName: \"kubernetes.io/projected/49a91781-5cb3-40c3-acba-91c9b1e807c4-kube-api-access-6l5qn\") pod \"multus-6nnwv\" (UID: \"49a91781-5cb3-40c3-acba-91c9b1e807c4\") " pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:08.821845 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.821828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs59j\" (UniqueName: \"kubernetes.io/projected/0eff9f94-0ca6-4609-8d05-bfc69a12f1f7-kube-api-access-xs59j\") pod \"ovnkube-node-sq668\" (UID: \"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.821981 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.821964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pm7\" (UniqueName: \"kubernetes.io/projected/efb3b2a5-0f80-4320-b493-a18d2917a8f3-kube-api-access-h9pm7\") pod \"multus-additional-cni-plugins-lgjs4\" (UID: \"efb3b2a5-0f80-4320-b493-a18d2917a8f3\") " pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:08.872657 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.872635    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f25fc1428d45d3789fd496fdfa89c.slice/crio-d7b293959a925f95ac7ba82d008d06e64398b1160ba3bb5f6c6a955b1eb1fd3e WatchSource:0}: Error finding container d7b293959a925f95ac7ba82d008d06e64398b1160ba3bb5f6c6a955b1eb1fd3e: Status 404 returned error can't find the container with id d7b293959a925f95ac7ba82d008d06e64398b1160ba3bb5f6c6a955b1eb1fd3e
Jul 30 03:45:08.879542 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.879513    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:45:08.885504 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.885486    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ee2cd84792d9409e5acf4090158792.slice/crio-6269f4083b64695e22d37109c1b2da0a40d5828881820ab350a5490c5cc00231 WatchSource:0}: Error finding container 6269f4083b64695e22d37109c1b2da0a40d5828881820ab350a5490c5cc00231: Status 404 returned error can't find the container with id 6269f4083b64695e22d37109c1b2da0a40d5828881820ab350a5490c5cc00231
Jul 30 03:45:08.926630 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.926609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:08.934847 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.934823    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3706bd53_387b_41be_8596_e51f050f920e.slice/crio-f4a09d490b4fec9fa2bdaddd9c125b0581e81d0d2dd2477f81a9939ecaaef834 WatchSource:0}: Error finding container f4a09d490b4fec9fa2bdaddd9c125b0581e81d0d2dd2477f81a9939ecaaef834: Status 404 returned error can't find the container with id f4a09d490b4fec9fa2bdaddd9c125b0581e81d0d2dd2477f81a9939ecaaef834
Jul 30 03:45:08.935805 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.935791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xv4fs"
Jul 30 03:45:08.942751 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.942732    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f6c619_1847_44a2_a21e_a20912d358cd.slice/crio-360e8154c67a49a49d7afb1c0f06a024ef543f223b00f2574a2a9b76f3161ea7 WatchSource:0}: Error finding container 360e8154c67a49a49d7afb1c0f06a024ef543f223b00f2574a2a9b76f3161ea7: Status 404 returned error can't find the container with id 360e8154c67a49a49d7afb1c0f06a024ef543f223b00f2574a2a9b76f3161ea7
Jul 30 03:45:08.952818 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.952800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cbzzj"
Jul 30 03:45:08.960380 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.960363    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28a9969_37d4_4242_a701_75478e599497.slice/crio-0b0f10c4540fa30b8d0faf889b41f0eacd468565c5454fa92771c4e70e20305a WatchSource:0}: Error finding container 0b0f10c4540fa30b8d0faf889b41f0eacd468565c5454fa92771c4e70e20305a: Status 404 returned error can't find the container with id 0b0f10c4540fa30b8d0faf889b41f0eacd468565c5454fa92771c4e70e20305a
Jul 30 03:45:08.972278 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.972244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9h697"
Jul 30 03:45:08.978066 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.978043    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:08.979190 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.979173    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d87d97_1e85_402c_a297_1d98cbaa8dba.slice/crio-d7ef01aa10c08b93badab8a674e123b1133e34aad4a1b2b0d84700d766a86403 WatchSource:0}: Error finding container d7ef01aa10c08b93badab8a674e123b1133e34aad4a1b2b0d84700d766a86403: Status 404 returned error can't find the container with id d7ef01aa10c08b93badab8a674e123b1133e34aad4a1b2b0d84700d766a86403
Jul 30 03:45:08.984982 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.984960    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff9f94_0ca6_4609_8d05_bfc69a12f1f7.slice/crio-ebb9165c527f6eb4573966f8de6d68f142988d49076570c59c6419ba6339bec4 WatchSource:0}: Error finding container ebb9165c527f6eb4573966f8de6d68f142988d49076570c59c6419ba6339bec4: Status 404 returned error can't find the container with id ebb9165c527f6eb4573966f8de6d68f142988d49076570c59c6419ba6339bec4
Jul 30 03:45:08.989293 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.989277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn"
Jul 30 03:45:08.992046 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:08.992034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrzfr"
Jul 30 03:45:08.997298 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.997279    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4999b8fa_fb60_4155_97ac_856262bd6ea3.slice/crio-b298ff6bb1518b24d9e0ddd4b78df406e14ef830414269091b8d11103b277139 WatchSource:0}: Error finding container b298ff6bb1518b24d9e0ddd4b78df406e14ef830414269091b8d11103b277139: Status 404 returned error can't find the container with id b298ff6bb1518b24d9e0ddd4b78df406e14ef830414269091b8d11103b277139
Jul 30 03:45:08.999607 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:08.999592    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35991ee7_db6e_4812_94fc_3d81199178b6.slice/crio-c0a9abd30707f607d60bc77440d4ba95d83a250e15fc4c279253b7a2dde59afc WatchSource:0}: Error finding container c0a9abd30707f607d60bc77440d4ba95d83a250e15fc4c279253b7a2dde59afc: Status 404 returned error can't find the container with id c0a9abd30707f607d60bc77440d4ba95d83a250e15fc4c279253b7a2dde59afc
Jul 30 03:45:09.023932 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.023914    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6nnwv"
Jul 30 03:45:09.032609 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.032589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgjs4"
Jul 30 03:45:09.032843 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:09.032822    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a91781_5cb3_40c3_acba_91c9b1e807c4.slice/crio-9c0e65066be029ec691f1c5f7e249a0ee1c4b17f473b91df7d91cbe20231d478 WatchSource:0}: Error finding container 9c0e65066be029ec691f1c5f7e249a0ee1c4b17f473b91df7d91cbe20231d478: Status 404 returned error can't find the container with id 9c0e65066be029ec691f1c5f7e249a0ee1c4b17f473b91df7d91cbe20231d478
Jul 30 03:45:09.040954 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:09.040935    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb3b2a5_0f80_4320_b493_a18d2917a8f3.slice/crio-c177b5a8fc61b2d75f0ef1af16878736b86ed0e981d3be28445f6a237cac77f6 WatchSource:0}: Error finding container c177b5a8fc61b2d75f0ef1af16878736b86ed0e981d3be28445f6a237cac77f6: Status 404 returned error can't find the container with id c177b5a8fc61b2d75f0ef1af16878736b86ed0e981d3be28445f6a237cac77f6
Jul 30 03:45:09.042259 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.042243    2562 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-lgjs4_openshift-multus(efb3b2a5-0f80-4320-b493-a18d2917a8f3): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 30 03:45:09.042308 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.042270    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" podUID="efb3b2a5-0f80-4320-b493-a18d2917a8f3"
Jul 30 03:45:09.215936 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.215886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:09.216023 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.216011    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:09.216069 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.216060    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:10.216043125 +0000 UTC m=+3.032979598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:09.243325 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.243306    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:09.316618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.316588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:09.316741 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.316712    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:09.316797 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.316778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:10.31675909 +0000 UTC m=+3.133695567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:09.374437 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.374407    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:09.418055 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.417922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:09.418189 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.418088    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:09.418189 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.418105    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:09.418189 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.418116    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:09.418189 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.418170    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:10.418153776 +0000 UTC m=+3.235090255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:09.646618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.646558    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:40:08 +0000 UTC" deadline="2028-02-15 19:47:23.558824568 +0000 UTC"
Jul 30 03:45:09.646618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.646615    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13576h2m13.912212992s"
Jul 30 03:45:09.796036 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.796007    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:09.796198 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.796147    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:09.810450 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.810400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nnwv" event={"ID":"49a91781-5cb3-40c3-acba-91c9b1e807c4","Type":"ContainerStarted","Data":"9c0e65066be029ec691f1c5f7e249a0ee1c4b17f473b91df7d91cbe20231d478"}
Jul 30 03:45:09.816603 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.816566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrzfr" event={"ID":"35991ee7-db6e-4812-94fc-3d81199178b6","Type":"ContainerStarted","Data":"c0a9abd30707f607d60bc77440d4ba95d83a250e15fc4c279253b7a2dde59afc"}
Jul 30 03:45:09.820824 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.820801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn" event={"ID":"4999b8fa-fb60-4155-97ac-856262bd6ea3","Type":"ContainerStarted","Data":"b298ff6bb1518b24d9e0ddd4b78df406e14ef830414269091b8d11103b277139"}
Jul 30 03:45:09.826889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.826863    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 03:45:09.828411 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.828389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"ebb9165c527f6eb4573966f8de6d68f142988d49076570c59c6419ba6339bec4"}
Jul 30 03:45:09.831572 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.831550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cbzzj" event={"ID":"d28a9969-37d4-4242-a701-75478e599497","Type":"ContainerStarted","Data":"0b0f10c4540fa30b8d0faf889b41f0eacd468565c5454fa92771c4e70e20305a"}
Jul 30 03:45:09.838975 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.838938    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xv4fs" event={"ID":"23f6c619-1847-44a2-a21e-a20912d358cd","Type":"ContainerStarted","Data":"360e8154c67a49a49d7afb1c0f06a024ef543f223b00f2574a2a9b76f3161ea7"}
Jul 30 03:45:09.844884 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.844860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6tcn" event={"ID":"3706bd53-387b-41be-8596-e51f050f920e","Type":"ContainerStarted","Data":"f4a09d490b4fec9fa2bdaddd9c125b0581e81d0d2dd2477f81a9939ecaaef834"}
Jul 30 03:45:09.862147 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.862124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerStarted","Data":"c177b5a8fc61b2d75f0ef1af16878736b86ed0e981d3be28445f6a237cac77f6"}
Jul 30 03:45:09.865288 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:09.865260    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" podUID="efb3b2a5-0f80-4320-b493-a18d2917a8f3"
Jul 30 03:45:09.873170 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.873141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9h697" event={"ID":"b7d87d97-1e85-402c-a297-1d98cbaa8dba","Type":"ContainerStarted","Data":"d7ef01aa10c08b93badab8a674e123b1133e34aad4a1b2b0d84700d766a86403"}
Jul 30 03:45:09.881550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.881496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal" event={"ID":"28ee2cd84792d9409e5acf4090158792","Type":"ContainerStarted","Data":"6269f4083b64695e22d37109c1b2da0a40d5828881820ab350a5490c5cc00231"}
Jul 30 03:45:09.887150 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:09.887078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal" event={"ID":"1b1f25fc1428d45d3789fd496fdfa89c","Type":"ContainerStarted","Data":"d7b293959a925f95ac7ba82d008d06e64398b1160ba3bb5f6c6a955b1eb1fd3e"}
Jul 30 03:45:10.222955 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.222302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:10.222955 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.222476    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:10.222955 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.222549    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:12.222531412 +0000 UTC m=+5.039467880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:10.323053 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.322999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:10.323261 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.323184    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:10.323261 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.323245    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:12.323228457 +0000 UTC m=+5.140164925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:10.424236 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.424186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:10.424401 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.424385    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:10.424444 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.424417    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:10.424485 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.424450    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:10.424533 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.424503    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:12.424485922 +0000 UTC m=+5.241422380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:10.647157 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.646765    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 03:40:08 +0000 UTC" deadline="2028-02-22 07:55:16.890523339 +0000 UTC"
Jul 30 03:45:10.647157 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.646804    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13732h10m6.243723274s"
Jul 30 03:45:10.779629 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.779590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:10.779629 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:10.779625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:10.779924 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.779737    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:10.779924 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:10.779854    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:11.778996 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:11.778967    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:11.779431 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:11.779104    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:12.239474 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:12.239440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:12.239660 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.239641    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:12.239734 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.239721    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:16.239701759 +0000 UTC m=+9.056638228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:12.340623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:12.340508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:12.340798 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.340671    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:12.340798 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.340751    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:16.340731227 +0000 UTC m=+9.157667693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:12.441636 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:12.441561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:12.441829 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.441775    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:12.441829 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.441796    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:12.441829 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.441808    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:12.441981 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.441866    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:16.441847866 +0000 UTC m=+9.258784334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:12.779426 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:12.779394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:12.779902 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.779516    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:12.779902 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:12.779762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:12.779902 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:12.779847    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:13.784823 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:13.784794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:13.785310 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:13.784923    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:14.779131 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:14.779053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:14.779131 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:14.779087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:14.779426 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:14.779189    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:14.779426 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:14.779341    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:15.779786 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:15.779754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:15.780223 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:15.779899    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:16.276842 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:16.276681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:16.276842 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.276824    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:16.277087 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.276900    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:24.276882646 +0000 UTC m=+17.093819114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:16.377056 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:16.377019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:16.377274 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.377139    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:16.377274 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.377198    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:24.377180348 +0000 UTC m=+17.194116804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:16.478077 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:16.478044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:16.478248 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.478182    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:16.478248 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.478206    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:16.478248 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.478218    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:16.478390 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.478296    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:24.478277288 +0000 UTC m=+17.295213762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:16.779829 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:16.779640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:16.779829 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:16.779671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:16.779829 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.779802    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:16.780385 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:16.779885    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:17.779536 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:17.779501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:17.779720 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:17.779639    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:18.778734 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:18.778706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:18.778734 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:18.778720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:18.779128 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:18.778819    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:18.779128 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:18.778940    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:19.779511 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:19.779461    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:19.779936 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:19.779619    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:20.779534 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:20.779480    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:20.779948 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:20.779563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:20.779948 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:20.779675    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:20.779948 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:20.779777    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:21.779630 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:21.779587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:21.780035 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:21.779708    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:22.779256 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:22.779229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:22.779395 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:22.779230    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:22.779453 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:22.779317    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:22.779453 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:22.779410    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:23.778909 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:23.778879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:23.779312 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:23.779015    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:24.333985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:24.333946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:24.334161 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.334064    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:24.334161 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.334116    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:40.334103573 +0000 UTC m=+33.151040031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:24.434791 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:24.434758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:24.434951 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.434873    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:24.434951 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.434935    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:40.434918344 +0000 UTC m=+33.251854815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:24.535200 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:24.535168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:24.535361 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.535340    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:24.535410 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.535368    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:24.535410 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.535383    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:24.535470 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.535444    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:40.535424494 +0000 UTC m=+33.352360971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:24.779440 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:24.779412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:24.779867 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.779510    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:24.779867 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:24.779566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:24.779867 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:24.779674    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:25.779404 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:25.779374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:25.779669 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:25.779497    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:26.779485 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:26.779312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:26.779640 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:26.779319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:26.779640 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:26.779575    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:26.779640 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:26.779625    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:27.779639 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.779453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:27.779772 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:27.779747    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:27.928176 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.928137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6nnwv" event={"ID":"49a91781-5cb3-40c3-acba-91c9b1e807c4","Type":"ContainerStarted","Data":"09ad9b38575378dd5f1319c1277176200ca7a9d7ffd09766bfbd402b02d0256b"}
Jul 30 03:45:27.929687 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.929655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrzfr" event={"ID":"35991ee7-db6e-4812-94fc-3d81199178b6","Type":"ContainerStarted","Data":"0db6bc7ffc21d12e3f05eea2606230a69c97c57342a958d3a9d5d57953d0779d"}
Jul 30 03:45:27.931548 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.931492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn" event={"ID":"4999b8fa-fb60-4155-97ac-856262bd6ea3","Type":"ContainerStarted","Data":"9510456cca6fdacbc0d18eb75f26ba80244a5ba4711e2210cbd9098af1ef2a9c"}
Jul 30 03:45:27.940842 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.940818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"4aec4dc1c1b1479a47a258b0e484377b24922a3622e22d012e62ada3f17b1a14"}
Jul 30 03:45:27.940934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.940922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"3e22de66dfeaaac135eabdb0765ba5dade1714c0022b81618ee7acd9e21a3a69"}
Jul 30 03:45:27.940992 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.940968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"83a99a535aa777c2ebc05d80296d9202bd609505998ec79cec4e623bcb3079e8"}
Jul 30 03:45:27.941035 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.941013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"4a9b20b25e0cd055543a00a872016510e8883d3835fb46c4646c7a63577e89d9"}
Jul 30 03:45:27.941080 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.941056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"2b8fb355fbe8fc63851ee5f1c40b2bcd2f05802f06bc7e4d7416c8797e3ad70f"}
Jul 30 03:45:27.941129 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.941101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"bfe09501cc54ff9bf855c66224af4fecc6ea12d9be8a13858b07e3ef79d00141"}
Jul 30 03:45:27.943940 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.942593    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cbzzj" event={"ID":"d28a9969-37d4-4242-a701-75478e599497","Type":"ContainerStarted","Data":"d128551f07350ea8d92d298e5c9263176fcc54419efc7da69eb7cb3495633e6c"}
Jul 30 03:45:27.945259 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.945238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xv4fs" event={"ID":"23f6c619-1847-44a2-a21e-a20912d358cd","Type":"ContainerStarted","Data":"18b6628fba4c2157249aff491c444a3e712f8bb3000b4ead7a0fccca416ecf85"}
Jul 30 03:45:27.946275 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.946219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6nnwv" podStartSLOduration=2.707706294 podStartE2EDuration="20.946206213s" totalImagesPullingTime="18.238499919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:27.945974155 +0000 UTC m=+20.762910657" watchObservedRunningTime="2026-07-30 03:45:27.946206213 +0000 UTC m=+20.763142688"
Jul 30 03:45:27.946993 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.946968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v6tcn" event={"ID":"3706bd53-387b-41be-8596-e51f050f920e","Type":"ContainerStarted","Data":"57ab44c51402ecb6cb6989aecbc4fe520c54933076e6eeab3371151c7dbd6905"}
Jul 30 03:45:27.948325 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.948302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal" event={"ID":"28ee2cd84792d9409e5acf4090158792","Type":"ContainerStarted","Data":"8c207ab2e6f179107a820854d9ddd9756e65b27694bc97a5b01d311886d10483"}
Jul 30 03:45:27.949920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.949900    2562 generic.go:356] "Generic (PLEG): container finished" podID="1b1f25fc1428d45d3789fd496fdfa89c" containerID="76017462ad2e0222e7f7b84daa37042867aabf09d2bc5455868e814b86e88e68" exitCode=0
Jul 30 03:45:27.950031 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.949927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal" event={"ID":"1b1f25fc1428d45d3789fd496fdfa89c","Type":"ContainerDied","Data":"76017462ad2e0222e7f7b84daa37042867aabf09d2bc5455868e814b86e88e68"}
Jul 30 03:45:27.974031 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.973919    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zrzfr" podStartSLOduration=2.945548692 podStartE2EDuration="20.973905263s" totalImagesPullingTime="18.028356571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:27.959313462 +0000 UTC m=+20.776249939" watchObservedRunningTime="2026-07-30 03:45:27.973905263 +0000 UTC m=+20.790841738"
Jul 30 03:45:27.974254 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.974228    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xv4fs" podStartSLOduration=2.885352395 podStartE2EDuration="20.974220371s" totalImagesPullingTime="18.088867976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:27.974007045 +0000 UTC m=+20.790943520" watchObservedRunningTime="2026-07-30 03:45:27.974220371 +0000 UTC m=+20.791156847"
Jul 30 03:45:27.987621 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.987579    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cbzzj" podStartSLOduration=2.919654442 podStartE2EDuration="20.987569862s" totalImagesPullingTime="18.06791542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:27.986850646 +0000 UTC m=+20.803787136" watchObservedRunningTime="2026-07-30 03:45:27.987569862 +0000 UTC m=+20.804506339"
Jul 30 03:45:27.999685 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:27.999651    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-44.ec2.internal" podStartSLOduration=20.99962521 podStartE2EDuration="20.99962521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:27.999234061 +0000 UTC m=+20.816170539" watchObservedRunningTime="2026-07-30 03:45:27.99962521 +0000 UTC m=+20.816561685"
Jul 30 03:45:28.015766 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.015732    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v6tcn" podStartSLOduration=2.92141121 podStartE2EDuration="21.01572182s" totalImagesPullingTime="18.09431061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:28.015417726 +0000 UTC m=+20.832354203" watchObservedRunningTime="2026-07-30 03:45:28.01572182 +0000 UTC m=+20.832658295"
Jul 30 03:45:28.779320 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.779126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:28.779421 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:28.779401    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:28.779658 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.779175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:28.779658 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:28.779617    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:28.953073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.953038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal" event={"ID":"1b1f25fc1428d45d3789fd496fdfa89c","Type":"ContainerStarted","Data":"84fe94771329810d59feb837a145ecbdf04b1dd6f2de6876658504f5676c4ac7"}
Jul 30 03:45:28.954461 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.954438    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="1fc23f85673e6a37b3c5cb0dc37f2c5bf15f99c435827265bc4f202c93f7076e" exitCode=0
Jul 30 03:45:28.954565 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.954503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"1fc23f85673e6a37b3c5cb0dc37f2c5bf15f99c435827265bc4f202c93f7076e"}
Jul 30 03:45:28.955954 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.955929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9h697" event={"ID":"b7d87d97-1e85-402c-a297-1d98cbaa8dba","Type":"ContainerStarted","Data":"b3b69df4aa7fedc511bd7f01e8a77f67da66ceeb78a2c38cf2e5703c27f3fcc2"}
Jul 30 03:45:28.966824 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.966784    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-44.ec2.internal" podStartSLOduration=21.966772937 podStartE2EDuration="21.966772937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:28.966486519 +0000 UTC m=+21.783422994" watchObservedRunningTime="2026-07-30 03:45:28.966772937 +0000 UTC m=+21.783709412"
Jul 30 03:45:28.999375 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:28.999335    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9h697" podStartSLOduration=3.948524366 podStartE2EDuration="21.999323237s" totalImagesPullingTime="18.050798871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:28.980127101 +0000 UTC m=+21.797063576" watchObservedRunningTime="2026-07-30 03:45:28.999323237 +0000 UTC m=+21.816259726"
Jul 30 03:45:29.519474 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.519452    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 03:45:29.672137 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.671998    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T03:45:29.519469677Z","UUID":"0fd82756-5bc8-4a52-b9bc-e5eda0dac00d","Handler":null,"Name":"","Endpoint":""}
Jul 30 03:45:29.673466 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.673448    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 03:45:29.673588 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.673487    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 03:45:29.779604 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.779578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:29.779733 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:29.779701    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:29.961049 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.961013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn" event={"ID":"4999b8fa-fb60-4155-97ac-856262bd6ea3","Type":"ContainerStarted","Data":"3d7082b5028c002f45a6316d6a137525de1a52bd95753f39299f72c3a99e6a94"}
Jul 30 03:45:29.964742 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:29.964635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"b26e25034bdd4adbefca7430986d225ac76d80795be5a3141bb45e3de4cf1799"}
Jul 30 03:45:30.779559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:30.779481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:30.779559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:30.779512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:30.779749 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:30.779625    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:30.779796 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:30.779742    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:30.968470 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:30.968434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn" event={"ID":"4999b8fa-fb60-4155-97ac-856262bd6ea3","Type":"ContainerStarted","Data":"042b60006362edccae4cddf381ebe6df20f77bc592208e9f8fa6ae0779cfadef"}
Jul 30 03:45:30.986165 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:30.986126    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sk9gn" podStartSLOduration=3.138460303 podStartE2EDuration="23.986112977s" totalImagesPullingTime="20.847652674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:30.985587314 +0000 UTC m=+23.802523791" watchObservedRunningTime="2026-07-30 03:45:30.986112977 +0000 UTC m=+23.803049453"
Jul 30 03:45:31.191094 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:31.191016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:31.191694 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:31.191675    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:31.779613 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:31.779582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:31.779789 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:31.779695    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:31.970434 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:31.970318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:31.970741 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:31.970670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v6tcn"
Jul 30 03:45:32.779068 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:32.779040    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:32.779216 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:32.779159    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:32.779275 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:32.779214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:32.779335 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:32.779308    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:33.779504 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.779315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:33.780310 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:33.779633    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:33.977323 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.977280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" event={"ID":"0eff9f94-0ca6-4609-8d05-bfc69a12f1f7","Type":"ContainerStarted","Data":"782362531ef92dacb43cfd6518141a3263b9e101ce85b794c67c8e7b2c0bfedd"}
Jul 30 03:45:33.977594 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.977568    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:33.977708 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.977688    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:33.977773 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.977733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:33.979159 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.979132    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="102468e675aed8cb991133eaecfaf37ed2289855aaeaca9f1defa65e2fcab8f7" exitCode=0
Jul 30 03:45:33.979267 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.979200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"102468e675aed8cb991133eaecfaf37ed2289855aaeaca9f1defa65e2fcab8f7"}
Jul 30 03:45:33.992556 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.992533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:33.992653 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:33.992639    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:45:34.002510 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:34.002474    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sq668" podStartSLOduration=8.908571016 podStartE2EDuration="27.002460807s" totalImagesPullingTime="18.093889791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:34.002440952 +0000 UTC m=+26.819377429" watchObservedRunningTime="2026-07-30 03:45:34.002460807 +0000 UTC m=+26.819397275"
Jul 30 03:45:34.778811 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:34.778779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:34.779036 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:34.778780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:34.779036 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:34.778889    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:34.779036 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:34.778992    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:35.151509 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.151484    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wfpd9"]
Jul 30 03:45:35.151871 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.151591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:35.151871 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:35.151689    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:35.154735 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.154713    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s5n46"]
Jul 30 03:45:35.154844 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.154805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:35.154939 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:35.154902    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:35.155356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.155333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svjms"]
Jul 30 03:45:35.155446 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.155435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:35.155585 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:35.155545    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:35.985140 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.985115    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="0d17475f53b819ff3c7afe13770c6cef81329c2255d2023e069ebcf9f56483f2" exitCode=0
Jul 30 03:45:35.985282 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:35.985184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"0d17475f53b819ff3c7afe13770c6cef81329c2255d2023e069ebcf9f56483f2"}
Jul 30 03:45:36.778955 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:36.778928    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:36.779417 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:36.779029    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:36.779417 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:36.779035    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:36.779417 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:36.779066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:36.779417 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:36.779130    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:36.779417 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:36.779180    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:37.990150 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:37.990119    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="184c68697e8a795c458f65e2219da51993a0bf8d40c6d09a9b662575a0ccd4f2" exitCode=0
Jul 30 03:45:37.990486 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:37.990163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"184c68697e8a795c458f65e2219da51993a0bf8d40c6d09a9b662575a0ccd4f2"}
Jul 30 03:45:38.779706 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:38.779682    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:38.779877 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:38.779794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:38.779877 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:38.779797    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svjms" podUID="d6717a3e-b8b5-4bc5-bd06-bf26a1f41837"
Jul 30 03:45:38.779877 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:38.779813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:38.780041 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:38.779883    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wfpd9" podUID="8a1c3a26-fa64-4f10-a63f-8c56c0912d68"
Jul 30 03:45:38.780041 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:38.779965    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s5n46" podUID="a97357fe-93eb-48e3-9e7c-f8a49f900874"
Jul 30 03:45:40.074348 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.074090    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-44.ec2.internal" event="NodeReady"
Jul 30 03:45:40.074838 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.074407    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 03:45:40.115572 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.115546    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mvbds"]
Jul 30 03:45:40.137598 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.137570    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6fhsw"]
Jul 30 03:45:40.137792 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.137768    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.140015 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.139990    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 03:45:40.140242 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.140227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 03:45:40.140374 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.140344    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s29r\""
Jul 30 03:45:40.151608 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.151589    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvbds"]
Jul 30 03:45:40.151710 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.151618    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fhsw"]
Jul 30 03:45:40.151710 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.151692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.154860 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.154814    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 03:45:40.154966 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.154857    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxgj5\""
Jul 30 03:45:40.154966 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.154886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 03:45:40.155120 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.155093    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 03:45:40.252077 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac597b2b-d087-4705-850e-c741a9a3c63e-config-volume\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.252077 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac597b2b-d087-4705-850e-c741a9a3c63e-tmp-dir\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.252291 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrgj\" (UniqueName: \"kubernetes.io/projected/76f692da-ea9a-4623-8ee6-20a20440f4d7-kube-api-access-4rrgj\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.252291 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhnm\" (UniqueName: \"kubernetes.io/projected/ac597b2b-d087-4705-850e-c741a9a3c63e-kube-api-access-zhhnm\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.252291 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76f692da-ea9a-4623-8ee6-20a20440f4d7-cert\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.252409 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.252307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.353375 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrgj\" (UniqueName: \"kubernetes.io/projected/76f692da-ea9a-4623-8ee6-20a20440f4d7-kube-api-access-4rrgj\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.353375 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhnm\" (UniqueName: \"kubernetes.io/projected/ac597b2b-d087-4705-850e-c741a9a3c63e-kube-api-access-zhhnm\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.353618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76f692da-ea9a-4623-8ee6-20a20440f4d7-cert\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.353618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.353618 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.353555    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:40.353618 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.353611    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:40.853592291 +0000 UTC m=+33.670528759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:40.353837 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac597b2b-d087-4705-850e-c741a9a3c63e-config-volume\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.353837 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac597b2b-d087-4705-850e-c741a9a3c63e-tmp-dir\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.353837 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.353740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:40.353974 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.353874    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:40.353974 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.353927    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs podName:d6717a3e-b8b5-4bc5-bd06-bf26a1f41837 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:12.353914115 +0000 UTC m=+65.170850582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs") pod "network-metrics-daemon-svjms" (UID: "d6717a3e-b8b5-4bc5-bd06-bf26a1f41837") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 03:45:40.354187 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.354165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac597b2b-d087-4705-850e-c741a9a3c63e-tmp-dir\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.354361 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.354330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac597b2b-d087-4705-850e-c741a9a3c63e-config-volume\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.357484 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.357466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76f692da-ea9a-4623-8ee6-20a20440f4d7-cert\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.361799 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.361780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrgj\" (UniqueName: \"kubernetes.io/projected/76f692da-ea9a-4623-8ee6-20a20440f4d7-kube-api-access-4rrgj\") pod \"ingress-canary-6fhsw\" (UID: \"76f692da-ea9a-4623-8ee6-20a20440f4d7\") " pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.362114 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.362095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhnm\" (UniqueName: \"kubernetes.io/projected/ac597b2b-d087-4705-850e-c741a9a3c63e-kube-api-access-zhhnm\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.454797 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.454754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:40.455006 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.454913    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:40.455006 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.454997    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret podName:8a1c3a26-fa64-4f10-a63f-8c56c0912d68 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:12.45497746 +0000 UTC m=+65.271913934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret") pod "global-pull-secret-syncer-wfpd9" (UID: "8a1c3a26-fa64-4f10-a63f-8c56c0912d68") : object "kube-system"/"original-pull-secret" not registered
Jul 30 03:45:40.461663 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.461644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fhsw"
Jul 30 03:45:40.555831 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.555797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:40.556009 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.555990    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 03:45:40.556075 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.556019    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 03:45:40.556075 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.556033    2562 projected.go:196] Error preparing data for projected volume kube-api-access-fbp99 for pod openshift-network-diagnostics/network-check-target-s5n46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:40.556168 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.556109    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99 podName:a97357fe-93eb-48e3-9e7c-f8a49f900874 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:12.556089971 +0000 UTC m=+65.373026447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbp99" (UniqueName: "kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99") pod "network-check-target-s5n46" (UID: "a97357fe-93eb-48e3-9e7c-f8a49f900874") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 03:45:40.778917 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.778887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:45:40.778917 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.778905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:45:40.778917 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.778918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:45:40.781445 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.781425    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 03:45:40.781583 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.781568    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 03:45:40.782985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.782730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzw94\""
Jul 30 03:45:40.782985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.782781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 03:45:40.782985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.782735    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmgxg\""
Jul 30 03:45:40.782985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.782822    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 03:45:40.857590 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:40.857565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:40.857710 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.857695    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:40.857775 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:40.857759    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:41.857740088 +0000 UTC m=+34.674676544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:41.517613 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.517580    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"]
Jul 30 03:45:41.550237 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.550207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"]
Jul 30 03:45:41.550389 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.550336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"
Jul 30 03:45:41.553648 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.553445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zzcjt\""
Jul 30 03:45:41.553648 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.553448    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:41.553648 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.553498    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:41.626207 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.626176    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"]
Jul 30 03:45:41.647991 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.647966    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"]
Jul 30 03:45:41.648145 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.648069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"
Jul 30 03:45:41.650775 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.650753    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tsb7c\""
Jul 30 03:45:41.665741 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.664104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkdn\" (UniqueName: \"kubernetes.io/projected/c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db-kube-api-access-6vkdn\") pod \"volume-data-source-validator-549c6d54cf-d9jfp\" (UID: \"c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"
Jul 30 03:45:41.672038 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.672015    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"]
Jul 30 03:45:41.672155 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.672132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.674338 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.674318    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 30 03:45:41.674516 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.674496    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7htxp\""
Jul 30 03:45:41.674638 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.674560    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:41.674638 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.674568    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:41.686171 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.686147    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"]
Jul 30 03:45:41.686280 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.686228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"]
Jul 30 03:45:41.686280 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.686150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.686280 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.686260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"]
Jul 30 03:45:41.688665 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.688541    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 30 03:45:41.688797 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.688674    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4mxpp\""
Jul 30 03:45:41.688896 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.688881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:41.688960 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.688908    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 30 03:45:41.689269 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.689249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:41.726583 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.726554    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"]
Jul 30 03:45:41.742134 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.742106    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77b9dc4d85-dz8bz"]
Jul 30 03:45:41.742268 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.742234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.744932 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.744910    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 30 03:45:41.745510 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.745329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 30 03:45:41.745510 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.745437    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 30 03:45:41.745849 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.745829    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 30 03:45:41.746015 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.745995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8sbkg\""
Jul 30 03:45:41.756213 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.756194    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:45:41.756330 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.756315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.761788 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.761771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 30 03:45:41.762617 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.762595    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 30 03:45:41.762709 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.762633    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ttpv2\""
Jul 30 03:45:41.762709 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.762682    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 03:45:41.762809 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.762733    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 30 03:45:41.763077 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.763060    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 03:45:41.764575 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55773a70-3b09-4801-82a8-e1dee6bdf633-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.764678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlmc\" (UniqueName: \"kubernetes.io/projected/55773a70-3b09-4801-82a8-e1dee6bdf633-kube-api-access-5hlmc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.764678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkdn\" (UniqueName: \"kubernetes.io/projected/c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db-kube-api-access-6vkdn\") pod \"volume-data-source-validator-549c6d54cf-d9jfp\" (UID: \"c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"
Jul 30 03:45:41.764678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764633    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 30 03:45:41.764678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.764852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ps9f\" (UniqueName: \"kubernetes.io/projected/2064eab3-7345-45cf-87d5-9dc0edf41034-kube-api-access-2ps9f\") pod \"network-check-source-7d88cb55d8-vdg7f\" (UID: \"2064eab3-7345-45cf-87d5-9dc0edf41034\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"
Jul 30 03:45:41.764852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55773a70-3b09-4801-82a8-e1dee6bdf633-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.764852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.764828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hz8\" (UniqueName: \"kubernetes.io/projected/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-kube-api-access-92hz8\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.774295 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.774252    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"]
Jul 30 03:45:41.774295 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.774282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77b9dc4d85-dz8bz"]
Jul 30 03:45:41.774439 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.774298    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:45:41.774439 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.774376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.777771 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.777745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 03:45:41.777860 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.777745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 03:45:41.777860 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.777746    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 03:45:41.777971 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.777870    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tfs77\""
Jul 30 03:45:41.790222 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.790204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 03:45:41.792125 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.792106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkdn\" (UniqueName: \"kubernetes.io/projected/c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db-kube-api-access-6vkdn\") pod \"volume-data-source-validator-549c6d54cf-d9jfp\" (UID: \"c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"
Jul 30 03:45:41.860476 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.860451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"
Jul 30 03:45:41.865734 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlmc\" (UniqueName: \"kubernetes.io/projected/55773a70-3b09-4801-82a8-e1dee6bdf633-kube-api-access-5hlmc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.865796 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:41.865796 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-default-certificate\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.865889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ps9f\" (UniqueName: \"kubernetes.io/projected/2064eab3-7345-45cf-87d5-9dc0edf41034-kube-api-access-2ps9f\") pod \"network-check-source-7d88cb55d8-vdg7f\" (UID: \"2064eab3-7345-45cf-87d5-9dc0edf41034\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"
Jul 30 03:45:41.865889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.865889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.866034 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.865873    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:41.866034 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hz8\" (UniqueName: \"kubernetes.io/projected/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-kube-api-access-92hz8\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.866034 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.865938    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:43.86592233 +0000 UTC m=+36.682858784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:41.866034 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.865981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.866034 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55773a70-3b09-4801-82a8-e1dee6bdf633-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.866232 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.866445 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.866584 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbmx\" (UniqueName: \"kubernetes.io/projected/92912f38-2f66-4711-90e3-13fa6dada35c-kube-api-access-7hbmx\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.866584 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.866584 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866557    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92912f38-2f66-4711-90e3-13fa6dada35c-config\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.866746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.866746 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.866709    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:45:41.866843 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.866767    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls podName:ca8abfad-a40f-4f61-8367-2f1cf6f47a3f nodeName:}" failed. No retries permitted until 2026-07-30 03:45:42.366750865 +0000 UTC m=+35.183687324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls") pod "cluster-samples-operator-6748489656-wh6tq" (UID: "ca8abfad-a40f-4f61-8367-2f1cf6f47a3f") : secret "samples-operator-tls" not found
Jul 30 03:45:41.866843 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nht4r\" (UniqueName: \"kubernetes.io/projected/d1347232-d5a8-4c77-b0de-2609a00e9b05-kube-api-access-nht4r\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.866946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55773a70-3b09-4801-82a8-e1dee6bdf633-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.866946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-stats-auth\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.866946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.867083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.866954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.867083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.867000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz8j\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.867083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.867032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.867083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.867042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55773a70-3b09-4801-82a8-e1dee6bdf633-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.867083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.867068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92912f38-2f66-4711-90e3-13fa6dada35c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.869786 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.869700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55773a70-3b09-4801-82a8-e1dee6bdf633-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.874733 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.874712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hz8\" (UniqueName: \"kubernetes.io/projected/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-kube-api-access-92hz8\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:41.875491 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.875470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ps9f\" (UniqueName: \"kubernetes.io/projected/2064eab3-7345-45cf-87d5-9dc0edf41034-kube-api-access-2ps9f\") pod \"network-check-source-7d88cb55d8-vdg7f\" (UID: \"2064eab3-7345-45cf-87d5-9dc0edf41034\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"
Jul 30 03:45:41.876754 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.876737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlmc\" (UniqueName: \"kubernetes.io/projected/55773a70-3b09-4801-82a8-e1dee6bdf633-kube-api-access-5hlmc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t\" (UID: \"55773a70-3b09-4801-82a8-e1dee6bdf633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:41.958136 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.958104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"
Jul 30 03:45:41.968578 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.968699 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.968699 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.968699 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.968699 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbmx\" (UniqueName: \"kubernetes.io/projected/92912f38-2f66-4711-90e3-13fa6dada35c-kube-api-access-7hbmx\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92912f38-2f66-4711-90e3-13fa6dada35c-config\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nht4r\" (UniqueName: \"kubernetes.io/projected/d1347232-d5a8-4c77-b0de-2609a00e9b05-kube-api-access-nht4r\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-stats-auth\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.968891 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.969130 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.968899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.970020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.970127    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.970153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.970198    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:42.470179813 +0000 UTC m=+35.287116291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : secret "router-metrics-certs-default" not found
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.970897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/92912f38-2f66-4711-90e3-13fa6dada35c-config\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.971355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.971399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz8j\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.971427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.971457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92912f38-2f66-4711-90e3-13fa6dada35c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.971512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-default-certificate\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.972248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.972914    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-stats-auth\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.972934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.971971    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:42.471953726 +0000 UTC m=+35.288890197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.972257    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:45:41.973986 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.973495    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74946bc9c6-kmqw5: secret "image-registry-tls" not found
Jul 30 03:45:41.974839 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:41.973567    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls podName:be0bef51-e4c5-4cc4-a477-12ddba6ffe6e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:42.473549447 +0000 UTC m=+35.290485905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls") pod "image-registry-74946bc9c6-kmqw5" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e") : secret "image-registry-tls" not found
Jul 30 03:45:41.974839 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.974501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-default-certificate\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.975035 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.975014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92912f38-2f66-4711-90e3-13fa6dada35c-serving-cert\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.977212 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.977170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbmx\" (UniqueName: \"kubernetes.io/projected/92912f38-2f66-4711-90e3-13fa6dada35c-kube-api-access-7hbmx\") pod \"service-ca-operator-5d6c75f84f-tq785\" (UID: \"92912f38-2f66-4711-90e3-13fa6dada35c\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:41.977842 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.977822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nht4r\" (UniqueName: \"kubernetes.io/projected/d1347232-d5a8-4c77-b0de-2609a00e9b05-kube-api-access-nht4r\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:41.978343 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.978311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.982725 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.982703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz8j\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:41.995282 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:41.995135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"
Jul 30 03:45:42.037351 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.037274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fhsw"]
Jul 30 03:45:42.040202 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.040162    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp"]
Jul 30 03:45:42.040808 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:42.040708    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f692da_ea9a_4623_8ee6_20a20440f4d7.slice/crio-afd520dacb1ce9cca2201fa90e3b8e0d83e54c71b6d42ff2f34972e2d8c11dcc WatchSource:0}: Error finding container afd520dacb1ce9cca2201fa90e3b8e0d83e54c71b6d42ff2f34972e2d8c11dcc: Status 404 returned error can't find the container with id afd520dacb1ce9cca2201fa90e3b8e0d83e54c71b6d42ff2f34972e2d8c11dcc
Jul 30 03:45:42.042317 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:42.042267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66f94e4_e6c8_4c5c_9e2a_14b2f4b2d5db.slice/crio-99a942d287df2b0ee71daf0802a37038e7e4342a87cf85c6393272d75f5a987d WatchSource:0}: Error finding container 99a942d287df2b0ee71daf0802a37038e7e4342a87cf85c6393272d75f5a987d: Status 404 returned error can't find the container with id 99a942d287df2b0ee71daf0802a37038e7e4342a87cf85c6393272d75f5a987d
Jul 30 03:45:42.053029 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.053000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"
Jul 30 03:45:42.134843 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.134813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f"]
Jul 30 03:45:42.170774 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.170745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t"]
Jul 30 03:45:42.199064 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:42.199034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2064eab3_7345_45cf_87d5_9dc0edf41034.slice/crio-ee13a841f9a8e0b8603023e239346cc7ed0e21475668a5d9cf25135e89bb998e WatchSource:0}: Error finding container ee13a841f9a8e0b8603023e239346cc7ed0e21475668a5d9cf25135e89bb998e: Status 404 returned error can't find the container with id ee13a841f9a8e0b8603023e239346cc7ed0e21475668a5d9cf25135e89bb998e
Jul 30 03:45:42.199995 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:42.199961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55773a70_3b09_4801_82a8_e1dee6bdf633.slice/crio-d9168f63369fdd7c6e88ca04b40de1faffb6cf8c8e3734366329d71d25284aad WatchSource:0}: Error finding container d9168f63369fdd7c6e88ca04b40de1faffb6cf8c8e3734366329d71d25284aad: Status 404 returned error can't find the container with id d9168f63369fdd7c6e88ca04b40de1faffb6cf8c8e3734366329d71d25284aad
Jul 30 03:45:42.337108 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.336892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785"]
Jul 30 03:45:42.339835 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:42.339810    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92912f38_2f66_4711_90e3_13fa6dada35c.slice/crio-b5f11132eb8a31b08f39046472c1abc94ed225c30e681ed2ff3f8929a420b43b WatchSource:0}: Error finding container b5f11132eb8a31b08f39046472c1abc94ed225c30e681ed2ff3f8929a420b43b: Status 404 returned error can't find the container with id b5f11132eb8a31b08f39046472c1abc94ed225c30e681ed2ff3f8929a420b43b
Jul 30 03:45:42.375941 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.375922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:42.376074 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.376056    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:45:42.376120 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.376114    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls podName:ca8abfad-a40f-4f61-8367-2f1cf6f47a3f nodeName:}" failed. No retries permitted until 2026-07-30 03:45:43.376097601 +0000 UTC m=+36.193034055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls") pod "cluster-samples-operator-6748489656-wh6tq" (UID: "ca8abfad-a40f-4f61-8367-2f1cf6f47a3f") : secret "samples-operator-tls" not found
Jul 30 03:45:42.477314 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.477290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:42.477409 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.477344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:42.477409 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:42.477371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:42.477481 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477432    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 30 03:45:42.477509 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477491    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:43.477475673 +0000 UTC m=+36.294412128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : secret "router-metrics-certs-default" not found
Jul 30 03:45:42.477509 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477496    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:45:42.477509 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477506    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74946bc9c6-kmqw5: secret "image-registry-tls" not found
Jul 30 03:45:42.477623 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477552    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:43.477529434 +0000 UTC m=+36.294465902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:42.477623 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:42.477576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls podName:be0bef51-e4c5-4cc4-a477-12ddba6ffe6e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:43.47756645 +0000 UTC m=+36.294502913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls") pod "image-registry-74946bc9c6-kmqw5" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e") : secret "image-registry-tls" not found
Jul 30 03:45:43.009118 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.009031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785" event={"ID":"92912f38-2f66-4711-90e3-13fa6dada35c","Type":"ContainerStarted","Data":"b5f11132eb8a31b08f39046472c1abc94ed225c30e681ed2ff3f8929a420b43b"}
Jul 30 03:45:43.011900 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.011830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t" event={"ID":"55773a70-3b09-4801-82a8-e1dee6bdf633","Type":"ContainerStarted","Data":"d9168f63369fdd7c6e88ca04b40de1faffb6cf8c8e3734366329d71d25284aad"}
Jul 30 03:45:43.014432 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.014322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f" event={"ID":"2064eab3-7345-45cf-87d5-9dc0edf41034","Type":"ContainerStarted","Data":"ee13a841f9a8e0b8603023e239346cc7ed0e21475668a5d9cf25135e89bb998e"}
Jul 30 03:45:43.016554 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.016488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fhsw" event={"ID":"76f692da-ea9a-4623-8ee6-20a20440f4d7","Type":"ContainerStarted","Data":"afd520dacb1ce9cca2201fa90e3b8e0d83e54c71b6d42ff2f34972e2d8c11dcc"}
Jul 30 03:45:43.023602 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.022706    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="db9c6075d82e4f220059bf6e49112e73e5bde51d73e3d39aa151414e7e6e1623" exitCode=0
Jul 30 03:45:43.023602 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.022783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"db9c6075d82e4f220059bf6e49112e73e5bde51d73e3d39aa151414e7e6e1623"}
Jul 30 03:45:43.027151 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.027089    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp" event={"ID":"c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db","Type":"ContainerStarted","Data":"99a942d287df2b0ee71daf0802a37038e7e4342a87cf85c6393272d75f5a987d"}
Jul 30 03:45:43.391694 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.391612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:43.392414 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.391976    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:45:43.392414 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.392049    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls podName:ca8abfad-a40f-4f61-8367-2f1cf6f47a3f nodeName:}" failed. No retries permitted until 2026-07-30 03:45:45.392028022 +0000 UTC m=+38.208964698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls") pod "cluster-samples-operator-6748489656-wh6tq" (UID: "ca8abfad-a40f-4f61-8367-2f1cf6f47a3f") : secret "samples-operator-tls" not found
Jul 30 03:45:43.494248 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.494210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:43.494379 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.494291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:43.494379 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.494334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:43.494545 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.494507    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:45:43.494545 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.494540    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74946bc9c6-kmqw5: secret "image-registry-tls" not found
Jul 30 03:45:43.494654 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.494598    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls podName:be0bef51-e4c5-4cc4-a477-12ddba6ffe6e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:45.494579887 +0000 UTC m=+38.311516355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls") pod "image-registry-74946bc9c6-kmqw5" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e") : secret "image-registry-tls" not found
Jul 30 03:45:43.495028 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.495008    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 30 03:45:43.495103 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.495062    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:45.495048111 +0000 UTC m=+38.311984582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : secret "router-metrics-certs-default" not found
Jul 30 03:45:43.495144 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.495125    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:45.495115039 +0000 UTC m=+38.312051509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:43.897573 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:43.897545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:43.898100 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.897700    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:43.898100 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:43.897778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:47.897749708 +0000 UTC m=+40.714686165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:44.038533 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:44.038475    2562 generic.go:356] "Generic (PLEG): container finished" podID="efb3b2a5-0f80-4320-b493-a18d2917a8f3" containerID="245b6b8c962ed5bf0f12755ac7d6aceaf89f1add897a831b6b10cd18d6acb077" exitCode=0
Jul 30 03:45:44.038958 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:44.038625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerDied","Data":"245b6b8c962ed5bf0f12755ac7d6aceaf89f1add897a831b6b10cd18d6acb077"}
Jul 30 03:45:44.043147 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:44.043082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp" event={"ID":"c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db","Type":"ContainerStarted","Data":"755faf68c1d0d8d46257ba4cceff6cb541638e29abb2a01b12b20ee51220f3ee"}
Jul 30 03:45:44.079311 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:44.079260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d9jfp" podStartSLOduration=1.2797252239999999 podStartE2EDuration="3.079240618s" totalImagesPullingTime="1.799515394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:44.078590569 +0000 UTC m=+36.895527046" watchObservedRunningTime="2026-07-30 03:45:44.079240618 +0000 UTC m=+36.896177096"
Jul 30 03:45:45.050026 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.049813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" event={"ID":"efb3b2a5-0f80-4320-b493-a18d2917a8f3","Type":"ContainerStarted","Data":"5f5292603447fa685c258b0f2990ad9961402469ca0c21735c2c266087cc00a8"}
Jul 30 03:45:45.075058 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.075011    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lgjs4" podStartSLOduration=4.879799284 podStartE2EDuration="38.074996333s" totalImagesPullingTime="33.195197049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-30 03:45:45.072645939 +0000 UTC m=+37.889582417" watchObservedRunningTime="2026-07-30 03:45:45.074996333 +0000 UTC m=+37.891932809"
Jul 30 03:45:45.414256 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.414183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:45.414389 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.414322    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:45:45.414436 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.414389    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls podName:ca8abfad-a40f-4f61-8367-2f1cf6f47a3f nodeName:}" failed. No retries permitted until 2026-07-30 03:45:49.41437168 +0000 UTC m=+42.231308151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls") pod "cluster-samples-operator-6748489656-wh6tq" (UID: "ca8abfad-a40f-4f61-8367-2f1cf6f47a3f") : secret "samples-operator-tls" not found
Jul 30 03:45:45.515158 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.515125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:45.515303 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.515187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:45.515303 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:45.515222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:45.515303 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515284    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 30 03:45:45.515438 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515338    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:45:45.515438 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515351    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74946bc9c6-kmqw5: secret "image-registry-tls" not found
Jul 30 03:45:45.515438 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515365    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:49.515342856 +0000 UTC m=+42.332279317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : secret "router-metrics-certs-default" not found
Jul 30 03:45:45.515438 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515380    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:49.515374384 +0000 UTC m=+42.332310839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:45.515438 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:45.515398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls podName:be0bef51-e4c5-4cc4-a477-12ddba6ffe6e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:49.515389652 +0000 UTC m=+42.332326107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls") pod "image-registry-74946bc9c6-kmqw5" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e") : secret "image-registry-tls" not found
Jul 30 03:45:46.501001 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.500968    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v2jpw"]
Jul 30 03:45:46.533159 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.533127    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v2jpw"]
Jul 30 03:45:46.533299 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.533264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.535920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.535893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 03:45:46.536073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.535978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 03:45:46.536073 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.536024    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbcm\""
Jul 30 03:45:46.536884 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.536864    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 03:45:46.536996 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.536917    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 03:45:46.727012 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.726978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e839a7c-89b6-4915-a518-16ecbd17e771-data-volume\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.727181 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.727054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.727181 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.727128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.727181 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.727168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kqm\" (UniqueName: \"kubernetes.io/projected/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-api-access-n9kqm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.727336 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.727236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e839a7c-89b6-4915-a518-16ecbd17e771-crio-socket\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828142 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828142 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kqm\" (UniqueName: \"kubernetes.io/projected/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-api-access-n9kqm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828142 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e839a7c-89b6-4915-a518-16ecbd17e771-crio-socket\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828400 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e839a7c-89b6-4915-a518-16ecbd17e771-data-volume\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828400 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:46.828216    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:45:46.828400 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:46.828303    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:47.328281578 +0000 UTC m=+40.145218045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : configmap "kube-rbac-proxy" not found
Jul 30 03:45:46.828400 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828400 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e839a7c-89b6-4915-a518-16ecbd17e771-crio-socket\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828600 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.828414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e839a7c-89b6-4915-a518-16ecbd17e771-data-volume\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:46.828600 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:46.828474    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:46.828600 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:46.828582    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:47.328562792 +0000 UTC m=+40.145499261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:46.839909 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:46.839882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kqm\" (UniqueName: \"kubernetes.io/projected/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-api-access-n9kqm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:47.332320 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:47.332250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:47.332440 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:47.332355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:47.332440 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.332404    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:47.332555 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.332478    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:48.332457311 +0000 UTC m=+41.149393778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:47.332555 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.332535    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:45:47.332660 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.332583    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:48.332568254 +0000 UTC m=+41.149504728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : configmap "kube-rbac-proxy" not found
Jul 30 03:45:47.937109 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:47.937078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:47.937462 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.937198    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:47.937462 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:47.937253    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:55.937235957 +0000 UTC m=+48.754172425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:48.058316 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.058279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f" event={"ID":"2064eab3-7345-45cf-87d5-9dc0edf41034","Type":"ContainerStarted","Data":"a876fec7f7a1907943cf059a232efd5aa971f8531f4b9d0e376a23f77dc0d9d0"}
Jul 30 03:45:48.060011 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.059877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fhsw" event={"ID":"76f692da-ea9a-4623-8ee6-20a20440f4d7","Type":"ContainerStarted","Data":"fc98c03cbbe8b18782e0ceb2bf024963fc066f5550a3bf4b0c1f918728ce7dd5"}
Jul 30 03:45:48.061437 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.061413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785" event={"ID":"92912f38-2f66-4711-90e3-13fa6dada35c","Type":"ContainerStarted","Data":"1f7eab1cbbc47cd5b5ff18486be9eae16051206edce79c5b6b39996964a430eb"}
Jul 30 03:45:48.062904 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.062879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t" event={"ID":"55773a70-3b09-4801-82a8-e1dee6bdf633","Type":"ContainerStarted","Data":"473057e0521335727313f2fe3ad919113d1fb9fe205729d123c3dbdc3f1ca13c"}
Jul 30 03:45:48.072867 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.072827    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vdg7f" podStartSLOduration=2.016811339 podStartE2EDuration="7.07281354s" totalImagesPullingTime="5.056002201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:48.071760525 +0000 UTC m=+40.888697003" watchObservedRunningTime="2026-07-30 03:45:48.07281354 +0000 UTC m=+40.889750016"
Jul 30 03:45:48.086966 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.086918    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t" podStartSLOduration=2.04146948 podStartE2EDuration="7.086906096s" totalImagesPullingTime="5.045436616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:48.086259406 +0000 UTC m=+40.903195882" watchObservedRunningTime="2026-07-30 03:45:48.086906096 +0000 UTC m=+40.903842571"
Jul 30 03:45:48.103089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.103041    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785" podStartSLOduration=2.178167458 podStartE2EDuration="7.103023657s" totalImagesPullingTime="4.924856199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:48.102804025 +0000 UTC m=+40.919740502" watchObservedRunningTime="2026-07-30 03:45:48.103023657 +0000 UTC m=+40.919960116"
Jul 30 03:45:48.119087 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.119042    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6fhsw" podStartSLOduration=2.896115999 podStartE2EDuration="8.119027227s" totalImagesPullingTime="5.222911228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:48.118971392 +0000 UTC m=+40.935907866" watchObservedRunningTime="2026-07-30 03:45:48.119027227 +0000 UTC m=+40.935963703"
Jul 30 03:45:48.282634 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.282607    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrzfr_35991ee7-db6e-4812-94fc-3d81199178b6/dns-node-resolver/0.log"
Jul 30 03:45:48.341494 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.341454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:48.341680 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:48.341571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:48.341680 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:48.341592    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:48.341680 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:48.341650    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:50.341634665 +0000 UTC m=+43.158571119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:48.341680 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:48.341666    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:45:48.341887 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:48.341717    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:50.341701992 +0000 UTC m=+43.158638453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : configmap "kube-rbac-proxy" not found
Jul 30 03:45:49.083017 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.082984    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cbzzj_d28a9969-37d4-4242-a701-75478e599497/node-ca/0.log"
Jul 30 03:45:49.455492 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.455456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:49.455665 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.455638    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 30 03:45:49.455719 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.455711    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls podName:ca8abfad-a40f-4f61-8367-2f1cf6f47a3f nodeName:}" failed. No retries permitted until 2026-07-30 03:45:57.455695026 +0000 UTC m=+50.272631483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls") pod "cluster-samples-operator-6748489656-wh6tq" (UID: "ca8abfad-a40f-4f61-8367-2f1cf6f47a3f") : secret "samples-operator-tls" not found
Jul 30 03:45:49.556300 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.556265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:49.556462 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.556332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:49.556462 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.556358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:49.556462 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556423    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 30 03:45:49.556688 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556486    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 03:45:49.556688 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556490    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:57.556469526 +0000 UTC m=+50.373405994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : secret "router-metrics-certs-default" not found
Jul 30 03:45:49.556688 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556497    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74946bc9c6-kmqw5: secret "image-registry-tls" not found
Jul 30 03:45:49.556688 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556510    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:57.55650097 +0000 UTC m=+50.373437425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:49.556688 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:49.556557    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls podName:be0bef51-e4c5-4cc4-a477-12ddba6ffe6e nodeName:}" failed. No retries permitted until 2026-07-30 03:45:57.556546033 +0000 UTC m=+50.373482491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls") pod "image-registry-74946bc9c6-kmqw5" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e") : secret "image-registry-tls" not found
Jul 30 03:45:49.681504 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:49.681434    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6fhsw_76f692da-ea9a-4623-8ee6-20a20440f4d7/serve-healthcheck-canary/0.log"
Jul 30 03:45:50.362068 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:50.362034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:50.362491 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:50.362098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:50.362491 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:50.362158    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 03:45:50.362491 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:50.362202    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:50.362491 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:50.362208    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:54.362195336 +0000 UTC m=+47.179131795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : configmap "kube-rbac-proxy" not found
Jul 30 03:45:50.362491 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:50.362276    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:45:54.362259861 +0000 UTC m=+47.179196325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:51.098826 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.098791    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z5xkb"]
Jul 30 03:45:51.127745 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.127713    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z5xkb"]
Jul 30 03:45:51.127897 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.127811    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.130436 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.130408    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 30 03:45:51.130744 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.130727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 30 03:45:51.131556 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.131536    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 30 03:45:51.131640 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.131556    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 30 03:45:51.131640 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.131544    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p8bqj\""
Jul 30 03:45:51.167760 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.167732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/678c2833-fd18-4fb2-93de-47743e087820-signing-key\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.167899 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.167785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslpn\" (UniqueName: \"kubernetes.io/projected/678c2833-fd18-4fb2-93de-47743e087820-kube-api-access-xslpn\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.167899 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.167813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/678c2833-fd18-4fb2-93de-47743e087820-signing-cabundle\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.268414 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.268383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/678c2833-fd18-4fb2-93de-47743e087820-signing-key\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.268599 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.268428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslpn\" (UniqueName: \"kubernetes.io/projected/678c2833-fd18-4fb2-93de-47743e087820-kube-api-access-xslpn\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.268599 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.268447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/678c2833-fd18-4fb2-93de-47743e087820-signing-cabundle\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.269132 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.269110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/678c2833-fd18-4fb2-93de-47743e087820-signing-cabundle\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.270881 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.270861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/678c2833-fd18-4fb2-93de-47743e087820-signing-key\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.276412 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.276393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslpn\" (UniqueName: \"kubernetes.io/projected/678c2833-fd18-4fb2-93de-47743e087820-kube-api-access-xslpn\") pod \"service-ca-658d7b7c78-z5xkb\" (UID: \"678c2833-fd18-4fb2-93de-47743e087820\") " pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.436567 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.436478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb"
Jul 30 03:45:51.577309 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:51.577280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z5xkb"]
Jul 30 03:45:51.579906 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:51.579877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678c2833_fd18_4fb2_93de_47743e087820.slice/crio-a8cd5586a5dd5ae251e9184a8e790dd57e64a8ccef7d045dd7febced360e7a61 WatchSource:0}: Error finding container a8cd5586a5dd5ae251e9184a8e790dd57e64a8ccef7d045dd7febced360e7a61: Status 404 returned error can't find the container with id a8cd5586a5dd5ae251e9184a8e790dd57e64a8ccef7d045dd7febced360e7a61
Jul 30 03:45:52.075390 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:52.075362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb" event={"ID":"678c2833-fd18-4fb2-93de-47743e087820","Type":"ContainerStarted","Data":"295b7e905a139cf6da9414b9761b89e7b8a3d123e99c0f16d8a56ad7f2a56fa2"}
Jul 30 03:45:52.075558 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:52.075409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb" event={"ID":"678c2833-fd18-4fb2-93de-47743e087820","Type":"ContainerStarted","Data":"a8cd5586a5dd5ae251e9184a8e790dd57e64a8ccef7d045dd7febced360e7a61"}
Jul 30 03:45:54.405652 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:54.405611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:54.406127 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:54.405682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:54.406127 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:54.405763    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:54.406127 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:54.405827    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:02.405810075 +0000 UTC m=+55.222746552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:45:54.406298 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:54.406158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e839a7c-89b6-4915-a518-16ecbd17e771-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:45:56.017474 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:56.017432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:45:56.017901 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:56.017610    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 03:45:56.017901 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:56.017670    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls podName:ac597b2b-d087-4705-850e-c741a9a3c63e nodeName:}" failed. No retries permitted until 2026-07-30 03:46:12.017655419 +0000 UTC m=+64.834591877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls") pod "dns-default-mvbds" (UID: "ac597b2b-d087-4705-850e-c741a9a3c63e") : secret "dns-default-metrics-tls" not found
Jul 30 03:45:57.528953 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.528923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:57.531010 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.530989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca8abfad-a40f-4f61-8367-2f1cf6f47a3f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wh6tq\" (UID: \"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:57.581498 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.581475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"
Jul 30 03:45:57.630229 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.629581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:57.630229 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.629665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:57.630229 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.629708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:57.630443 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:45:57.630302    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle podName:d1347232-d5a8-4c77-b0de-2609a00e9b05 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:13.630279238 +0000 UTC m=+66.447215711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle") pod "router-default-77b9dc4d85-dz8bz" (UID: "d1347232-d5a8-4c77-b0de-2609a00e9b05") : configmap references non-existent config key: service-ca.crt
Jul 30 03:45:57.632317 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.632278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"image-registry-74946bc9c6-kmqw5\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") " pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:57.632832 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.632809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1347232-d5a8-4c77-b0de-2609a00e9b05-metrics-certs\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:45:57.684117 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.684088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:57.699623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.699551    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-z5xkb" podStartSLOduration=6.699514668 podStartE2EDuration="6.699514668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:52.090887553 +0000 UTC m=+44.907824031" watchObservedRunningTime="2026-07-30 03:45:57.699514668 +0000 UTC m=+50.516451145"
Jul 30 03:45:57.701095 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.701038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq"]
Jul 30 03:45:57.815420 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:57.815394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:45:57.817573 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:45:57.817544    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0bef51_e4c5_4cc4_a477_12ddba6ffe6e.slice/crio-9181e0b5998080328a908a4d55d2a2379c22e1e1c255fbce63836af77e0314c5 WatchSource:0}: Error finding container 9181e0b5998080328a908a4d55d2a2379c22e1e1c255fbce63836af77e0314c5: Status 404 returned error can't find the container with id 9181e0b5998080328a908a4d55d2a2379c22e1e1c255fbce63836af77e0314c5
Jul 30 03:45:58.094272 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:58.094187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq" event={"ID":"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f","Type":"ContainerStarted","Data":"8fdd411262f5e8c3655b73b03876450fb31ed2f6f0f16bd8411caa3aeae6baa2"}
Jul 30 03:45:58.095563 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:58.095532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" event={"ID":"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e","Type":"ContainerStarted","Data":"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"}
Jul 30 03:45:58.095678 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:58.095577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" event={"ID":"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e","Type":"ContainerStarted","Data":"9181e0b5998080328a908a4d55d2a2379c22e1e1c255fbce63836af77e0314c5"}
Jul 30 03:45:58.095722 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:58.095676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:45:58.120037 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:45:58.119982    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" podStartSLOduration=17.119964232 podStartE2EDuration="17.119964232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:45:58.119376732 +0000 UTC m=+50.936313211" watchObservedRunningTime="2026-07-30 03:45:58.119964232 +0000 UTC m=+50.936900710"
Jul 30 03:46:00.104615 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:00.104585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq" event={"ID":"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f","Type":"ContainerStarted","Data":"1d7c59e7604984e3b404dead7deec4d2a28e927c00891f5641fc9609e57cc8e7"}
Jul 30 03:46:00.105040 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:00.104636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq" event={"ID":"ca8abfad-a40f-4f61-8367-2f1cf6f47a3f","Type":"ContainerStarted","Data":"e611c27419c792e6342c3c341228454f2a35167ee5b7d99e925ffa443661f4f7"}
Jul 30 03:46:00.120841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:00.120799    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wh6tq" podStartSLOduration=17.267610681 podStartE2EDuration="19.120784406s" totalImagesPullingTime="1.853173725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:00.119442193 +0000 UTC m=+52.936378669" watchObservedRunningTime="2026-07-30 03:46:00.120784406 +0000 UTC m=+52.937720881"
Jul 30 03:46:02.471381 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:02.471339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:46:02.471871 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:46:02.471486    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 03:46:02.471871 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:46:02.471564    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls podName:0e839a7c-89b6-4915-a518-16ecbd17e771 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:18.471547517 +0000 UTC m=+71.288483971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v2jpw" (UID: "0e839a7c-89b6-4915-a518-16ecbd17e771") : secret "insights-runtime-extractor-tls" not found
Jul 30 03:46:05.996345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:05.996315    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq668"
Jul 30 03:46:09.829728 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.829680    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"]
Jul 30 03:46:09.836297 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.836273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.838984 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.838963    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 03:46:09.839095 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.839009    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 30 03:46:09.839095 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.838969    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 30 03:46:09.839217 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.839135    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 30 03:46:09.840076 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.840058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 03:46:09.840330 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.840314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 30 03:46:09.840374 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.840331    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 03:46:09.847738 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.847720    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"]
Jul 30 03:46:09.895579 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.895542    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:46:09.927855 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.927831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.927967 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.927861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.927967 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.927886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.927967 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.927928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nbp\" (UniqueName: \"kubernetes.io/projected/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-kube-api-access-k4nbp\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.928065 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.927993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.928065 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.928009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:09.936093 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.936064    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f7b4dff4f-ddcfr"]
Jul 30 03:46:09.939847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.939830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:09.952801 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:09.952777    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f7b4dff4f-ddcfr"]
Jul 30 03:46:10.029123 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-registry-tls\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029123 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nbp\" (UniqueName: \"kubernetes.io/projected/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-kube-api-access-k4nbp\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-trusted-ca\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bjv\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-kube-api-access-r7bjv\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-registry-certificates\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-bound-sa-token\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029334 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.029559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.029559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-installation-pull-secrets\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029469    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-image-registry-private-configuration\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029691 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf349c0-f872-405e-b222-a80397d52f0f-ca-trust-extracted\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.029691 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.029691 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.029629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.030372 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.030339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.031632 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.031602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-ca\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.031841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.031823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.031922 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.031874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.032101 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.032082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-hub\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.040178 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.040158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nbp\" (UniqueName: \"kubernetes.io/projected/200c1a1c-ecd8-4c3d-a284-8b0f0325d918-kube-api-access-k4nbp\") pod \"cluster-proxy-proxy-agent-86497bf-n9dhv\" (UID: \"200c1a1c-ecd8-4c3d-a284-8b0f0325d918\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.130277 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bjv\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-kube-api-access-r7bjv\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130277 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-registry-certificates\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130478 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-bound-sa-token\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130478 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-installation-pull-secrets\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130478 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-image-registry-private-configuration\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130662 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf349c0-f872-405e-b222-a80397d52f0f-ca-trust-extracted\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130662 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-registry-tls\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130760 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-trusted-ca\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.130927 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.130903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf349c0-f872-405e-b222-a80397d52f0f-ca-trust-extracted\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.131140 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.131115    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-registry-certificates\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.131609 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.131593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf349c0-f872-405e-b222-a80397d52f0f-trusted-ca\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.132918 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.132899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-registry-tls\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.133157 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.133140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-image-registry-private-configuration\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.133226 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.133205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf349c0-f872-405e-b222-a80397d52f0f-installation-pull-secrets\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.142146 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.142123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-bound-sa-token\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.142362 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.142347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bjv\" (UniqueName: \"kubernetes.io/projected/acf349c0-f872-405e-b222-a80397d52f0f-kube-api-access-r7bjv\") pod \"image-registry-f7b4dff4f-ddcfr\" (UID: \"acf349c0-f872-405e-b222-a80397d52f0f\") " pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.153292 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.153270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"
Jul 30 03:46:10.248249 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.248228    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:10.271654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.271633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv"]
Jul 30 03:46:10.273792 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:10.273762    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod200c1a1c_ecd8_4c3d_a284_8b0f0325d918.slice/crio-ac7f49d2d7a69193eadeb123bc738d9f1131615394e922a33d0772b619fb4ba1 WatchSource:0}: Error finding container ac7f49d2d7a69193eadeb123bc738d9f1131615394e922a33d0772b619fb4ba1: Status 404 returned error can't find the container with id ac7f49d2d7a69193eadeb123bc738d9f1131615394e922a33d0772b619fb4ba1
Jul 30 03:46:10.374183 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:10.374122    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f7b4dff4f-ddcfr"]
Jul 30 03:46:10.376408 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:10.376384    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf349c0_f872_405e_b222_a80397d52f0f.slice/crio-f64630ae0e670785f3651d853b28bb6c417e565270d900bd3f99152504ca10a2 WatchSource:0}: Error finding container f64630ae0e670785f3651d853b28bb6c417e565270d900bd3f99152504ca10a2: Status 404 returned error can't find the container with id f64630ae0e670785f3651d853b28bb6c417e565270d900bd3f99152504ca10a2
Jul 30 03:46:11.139939 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:11.139899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv" event={"ID":"200c1a1c-ecd8-4c3d-a284-8b0f0325d918","Type":"ContainerStarted","Data":"ac7f49d2d7a69193eadeb123bc738d9f1131615394e922a33d0772b619fb4ba1"}
Jul 30 03:46:11.142083 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:11.142054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr" event={"ID":"acf349c0-f872-405e-b222-a80397d52f0f","Type":"ContainerStarted","Data":"97981bc8580922d89dbef902ef91aa324b86e2b8a5395af255da938cdef78c99"}
Jul 30 03:46:11.142196 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:11.142104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr" event={"ID":"acf349c0-f872-405e-b222-a80397d52f0f","Type":"ContainerStarted","Data":"f64630ae0e670785f3651d853b28bb6c417e565270d900bd3f99152504ca10a2"}
Jul 30 03:46:11.142196 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:11.142185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:11.162556 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:11.162494    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr" podStartSLOduration=2.162477483 podStartE2EDuration="2.162477483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:11.159923201 +0000 UTC m=+63.976859677" watchObservedRunningTime="2026-07-30 03:46:11.162477483 +0000 UTC m=+63.979413959"
Jul 30 03:46:12.050946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.050915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:46:12.053427 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.053394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac597b2b-d087-4705-850e-c741a9a3c63e-metrics-tls\") pod \"dns-default-mvbds\" (UID: \"ac597b2b-d087-4705-850e-c741a9a3c63e\") " pod="openshift-dns/dns-default-mvbds"
Jul 30 03:46:12.251227 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.251200    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5s29r\""
Jul 30 03:46:12.259307 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.259283    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvbds"
Jul 30 03:46:12.454857 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.454773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:46:12.457494 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.457470    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 03:46:12.467926 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.467904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6717a3e-b8b5-4bc5-bd06-bf26a1f41837-metrics-certs\") pod \"network-metrics-daemon-svjms\" (UID: \"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837\") " pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:46:12.556746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.556103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:46:12.567220 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.567196    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 03:46:12.578962 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.578934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8a1c3a26-fa64-4f10-a63f-8c56c0912d68-original-pull-secret\") pod \"global-pull-secret-syncer-wfpd9\" (UID: \"8a1c3a26-fa64-4f10-a63f-8c56c0912d68\") " pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:46:12.592706 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.592683    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kzw94\""
Jul 30 03:46:12.597085 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.597065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wfpd9"
Jul 30 03:46:12.601141 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.601103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svjms"
Jul 30 03:46:12.605915 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.605892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvbds"]
Jul 30 03:46:12.611323 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:12.611299    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac597b2b_d087_4705_850e_c741a9a3c63e.slice/crio-62c1a7179d4d28cd5e05a11cad16fc37768e7e898e51389fab67161df2c6e238 WatchSource:0}: Error finding container 62c1a7179d4d28cd5e05a11cad16fc37768e7e898e51389fab67161df2c6e238: Status 404 returned error can't find the container with id 62c1a7179d4d28cd5e05a11cad16fc37768e7e898e51389fab67161df2c6e238
Jul 30 03:46:12.657550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.657152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:46:12.663122 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.663071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbp99\" (UniqueName: \"kubernetes.io/projected/a97357fe-93eb-48e3-9e7c-f8a49f900874-kube-api-access-fbp99\") pod \"network-check-target-s5n46\" (UID: \"a97357fe-93eb-48e3-9e7c-f8a49f900874\") " pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:46:12.741629 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.741592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wfpd9"]
Jul 30 03:46:12.744904 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:12.744873    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1c3a26_fa64_4f10_a63f_8c56c0912d68.slice/crio-877122e1d9ef261793abc10e06d484fc7231c3241fad2ba389cae4cb00386755 WatchSource:0}: Error finding container 877122e1d9ef261793abc10e06d484fc7231c3241fad2ba389cae4cb00386755: Status 404 returned error can't find the container with id 877122e1d9ef261793abc10e06d484fc7231c3241fad2ba389cae4cb00386755
Jul 30 03:46:12.763650 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.763620    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svjms"]
Jul 30 03:46:12.765987 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:12.765960    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6717a3e_b8b5_4bc5_bd06_bf26a1f41837.slice/crio-f6f4aa052c48f4cb2b58fc4900a91e35a3ffc45a3f7909c02e1e154ae39a9394 WatchSource:0}: Error finding container f6f4aa052c48f4cb2b58fc4900a91e35a3ffc45a3f7909c02e1e154ae39a9394: Status 404 returned error can't find the container with id f6f4aa052c48f4cb2b58fc4900a91e35a3ffc45a3f7909c02e1e154ae39a9394
Jul 30 03:46:12.905756 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.905729    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmgxg\""
Jul 30 03:46:12.913852 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:12.913829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:46:13.074726 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.074700    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s5n46"]
Jul 30 03:46:13.154010 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.153943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svjms" event={"ID":"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837","Type":"ContainerStarted","Data":"f6f4aa052c48f4cb2b58fc4900a91e35a3ffc45a3f7909c02e1e154ae39a9394"}
Jul 30 03:46:13.155429 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.155402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wfpd9" event={"ID":"8a1c3a26-fa64-4f10-a63f-8c56c0912d68","Type":"ContainerStarted","Data":"877122e1d9ef261793abc10e06d484fc7231c3241fad2ba389cae4cb00386755"}
Jul 30 03:46:13.156572 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.156545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvbds" event={"ID":"ac597b2b-d087-4705-850e-c741a9a3c63e","Type":"ContainerStarted","Data":"62c1a7179d4d28cd5e05a11cad16fc37768e7e898e51389fab67161df2c6e238"}
Jul 30 03:46:13.666072 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.666038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:13.666706 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.666687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1347232-d5a8-4c77-b0de-2609a00e9b05-service-ca-bundle\") pod \"router-default-77b9dc4d85-dz8bz\" (UID: \"d1347232-d5a8-4c77-b0de-2609a00e9b05\") " pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:13.867160 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.867129    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ttpv2\""
Jul 30 03:46:13.875119 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:13.875087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:14.613243 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:14.613211    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97357fe_93eb_48e3_9e7c_f8a49f900874.slice/crio-fe61fac5b93f13d400025d18fde3b73dfcbccf96cad24d43e207fe615543db2c WatchSource:0}: Error finding container fe61fac5b93f13d400025d18fde3b73dfcbccf96cad24d43e207fe615543db2c: Status 404 returned error can't find the container with id fe61fac5b93f13d400025d18fde3b73dfcbccf96cad24d43e207fe615543db2c
Jul 30 03:46:15.060377 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.060303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77b9dc4d85-dz8bz"]
Jul 30 03:46:15.065021 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:15.064992    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1347232_d5a8_4c77_b0de_2609a00e9b05.slice/crio-180a6f0417ef798a94641c56597c5210337d4737fed9b8303d97c6f1d0f19637 WatchSource:0}: Error finding container 180a6f0417ef798a94641c56597c5210337d4737fed9b8303d97c6f1d0f19637: Status 404 returned error can't find the container with id 180a6f0417ef798a94641c56597c5210337d4737fed9b8303d97c6f1d0f19637
Jul 30 03:46:15.175069 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.173688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz" event={"ID":"d1347232-d5a8-4c77-b0de-2609a00e9b05","Type":"ContainerStarted","Data":"61e717dd6b250e3acd2ab320983e5b766c6257d8b92b1b130db86645d235ecb5"}
Jul 30 03:46:15.175069 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.173782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz" event={"ID":"d1347232-d5a8-4c77-b0de-2609a00e9b05","Type":"ContainerStarted","Data":"180a6f0417ef798a94641c56597c5210337d4737fed9b8303d97c6f1d0f19637"}
Jul 30 03:46:15.176455 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.176196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvbds" event={"ID":"ac597b2b-d087-4705-850e-c741a9a3c63e","Type":"ContainerStarted","Data":"f7c3162a3cf3cb522eb4948d19b9e1797463e25892274c1cf236bf39b544e44b"}
Jul 30 03:46:15.178737 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.178609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s5n46" event={"ID":"a97357fe-93eb-48e3-9e7c-f8a49f900874","Type":"ContainerStarted","Data":"f9810c5c5f3fe8c7c75b38386749bfc3d02c904850a42f6902e70f7292a68c54"}
Jul 30 03:46:15.178737 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.178653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s5n46" event={"ID":"a97357fe-93eb-48e3-9e7c-f8a49f900874","Type":"ContainerStarted","Data":"fe61fac5b93f13d400025d18fde3b73dfcbccf96cad24d43e207fe615543db2c"}
Jul 30 03:46:15.179067 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.179051    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:46:15.184638 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.184563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv" event={"ID":"200c1a1c-ecd8-4c3d-a284-8b0f0325d918","Type":"ContainerStarted","Data":"7b70ca72d541214d4b24f4138fbb8d8fa196ff1ce9c67872062f3d3857f560b6"}
Jul 30 03:46:15.184638 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.184600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv" event={"ID":"200c1a1c-ecd8-4c3d-a284-8b0f0325d918","Type":"ContainerStarted","Data":"6cf4bcccec391953d90f0456db6ef23f3b6282e255eba7ba424d5b848c90e1da"}
Jul 30 03:46:15.187802 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.187743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svjms" event={"ID":"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837","Type":"ContainerStarted","Data":"acc2477b19524a2f65d46cfd87758d03caa67e7d1fc60285078763659c72f40a"}
Jul 30 03:46:15.192855 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.192804    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz" podStartSLOduration=34.192786162 podStartE2EDuration="34.192786162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:15.19226121 +0000 UTC m=+68.009197697" watchObservedRunningTime="2026-07-30 03:46:15.192786162 +0000 UTC m=+68.009722640"
Jul 30 03:46:15.211805 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.210649    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s5n46" podStartSLOduration=68.210638804 podStartE2EDuration="1m8.210638804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:15.210304515 +0000 UTC m=+68.027241005" watchObservedRunningTime="2026-07-30 03:46:15.210638804 +0000 UTC m=+68.027575280"
Jul 30 03:46:15.875714 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.875679    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:15.878644 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:15.878617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:16.194340 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.194259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv" event={"ID":"200c1a1c-ecd8-4c3d-a284-8b0f0325d918","Type":"ContainerStarted","Data":"7507347ce0042ef5b52f2e3bc9f83f5c5bb0383c409519abcac6be02ab6e2555"}
Jul 30 03:46:16.196246 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.196216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svjms" event={"ID":"d6717a3e-b8b5-4bc5-bd06-bf26a1f41837","Type":"ContainerStarted","Data":"b86743f379d80b101360441b96a02e4e2abd2af553be3430e0d4dfbde1603ba3"}
Jul 30 03:46:16.198366 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.198345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvbds" event={"ID":"ac597b2b-d087-4705-850e-c741a9a3c63e","Type":"ContainerStarted","Data":"ea5825bffbecf1a1f44b21d957e2e370a48816f1756be6bd61604977d7ad5604"}
Jul 30 03:46:16.198886 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.198638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:16.199930 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.199908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77b9dc4d85-dz8bz"
Jul 30 03:46:16.212696 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.212637    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86497bf-n9dhv" podStartSLOduration=2.587448738 podStartE2EDuration="7.212624572s" totalImagesPullingTime="4.625175834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:16.211717877 +0000 UTC m=+69.028654367" watchObservedRunningTime="2026-07-30 03:46:16.212624572 +0000 UTC m=+69.029561072"
Jul 30 03:46:16.227940 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.227895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-svjms" podStartSLOduration=67.012633375 podStartE2EDuration="1m9.227882746s" totalImagesPullingTime="2.215249371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:16.226556238 +0000 UTC m=+69.043492716" watchObservedRunningTime="2026-07-30 03:46:16.227882746 +0000 UTC m=+69.044819220"
Jul 30 03:46:16.242635 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:16.242602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mvbds" podStartSLOduration=33.953658894 podStartE2EDuration="36.242590385s" totalImagesPullingTime="2.288931491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:16.242289886 +0000 UTC m=+69.059226361" watchObservedRunningTime="2026-07-30 03:46:16.242590385 +0000 UTC m=+69.059526861"
Jul 30 03:46:17.201958 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:17.201925    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mvbds"
Jul 30 03:46:18.505114 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:18.505087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:46:18.507162 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:18.507134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e839a7c-89b6-4915-a518-16ecbd17e771-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v2jpw\" (UID: \"0e839a7c-89b6-4915-a518-16ecbd17e771\") " pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:46:18.645948 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:18.645929    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbcm\""
Jul 30 03:46:18.653871 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:18.653854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v2jpw"
Jul 30 03:46:18.809703 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:18.809679    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v2jpw"]
Jul 30 03:46:18.814125 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:18.812576    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e839a7c_89b6_4915_a518_16ecbd17e771.slice/crio-0fb1d8ebfe56b736241a18c8f5e4186e1a29745cbaa40f8d3b0557a241c9287b WatchSource:0}: Error finding container 0fb1d8ebfe56b736241a18c8f5e4186e1a29745cbaa40f8d3b0557a241c9287b: Status 404 returned error can't find the container with id 0fb1d8ebfe56b736241a18c8f5e4186e1a29745cbaa40f8d3b0557a241c9287b
Jul 30 03:46:19.210666 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:19.210586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wfpd9" event={"ID":"8a1c3a26-fa64-4f10-a63f-8c56c0912d68","Type":"ContainerStarted","Data":"bd7ac7629e8de67cc482aab1b7297701bb2a735e0d296e401ee3e9e466ba0a56"}
Jul 30 03:46:19.211763 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:19.211737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2jpw" event={"ID":"0e839a7c-89b6-4915-a518-16ecbd17e771","Type":"ContainerStarted","Data":"0fb1d8ebfe56b736241a18c8f5e4186e1a29745cbaa40f8d3b0557a241c9287b"}
Jul 30 03:46:19.228868 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:19.228830    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wfpd9" podStartSLOduration=66.424656937 podStartE2EDuration="1m12.228815434s" totalImagesPullingTime="5.804158497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:19.22858766 +0000 UTC m=+72.045524136" watchObservedRunningTime="2026-07-30 03:46:19.228815434 +0000 UTC m=+72.045751910"
Jul 30 03:46:19.903430 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:19.903390    2562 patch_prober.go:28] interesting pod/image-registry-74946bc9c6-kmqw5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:46:19.903868 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:19.903457    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:46:24.231794 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:24.231764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2jpw" event={"ID":"0e839a7c-89b6-4915-a518-16ecbd17e771","Type":"ContainerStarted","Data":"11ed9f392faf912165303e9f9bf0280899171d902ee6a48356dce409ace41d99"}
Jul 30 03:46:27.208565 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:27.208540    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mvbds"
Jul 30 03:46:28.251735 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.251632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2jpw" event={"ID":"0e839a7c-89b6-4915-a518-16ecbd17e771","Type":"ContainerStarted","Data":"806b5904affdfc504244f3cc17007c6d278b8f19891be266d54e126bd7a82f8f"}
Jul 30 03:46:28.572774 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.572693    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-99blf"]
Jul 30 03:46:28.576033 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.576009    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.578490 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.578434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fg6jg\""
Jul 30 03:46:28.578763 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.578725    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 03:46:28.578808 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.578789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 03:46:28.579648 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.579607    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 03:46:28.579648 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.579617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 03:46:28.579799 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.579655    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 03:46:28.579934 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.579920    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 03:46:28.687315 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.687271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764kb\" (UniqueName: \"kubernetes.io/projected/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-kube-api-access-764kb\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.687710 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.687597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.687936 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.687854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-wtmp\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.687936 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.687901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.688319 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.688208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-metrics-client-ca\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.688458 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.688429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-sys\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.688702 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.688553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-root\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.688702 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.688585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.688702 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.688630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-textfile\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.789871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.789928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-wtmp\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.789962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.789989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-metrics-client-ca\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.790027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-sys\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.790047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-root\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.790091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.790134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-textfile\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.790178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764kb\" (UniqueName: \"kubernetes.io/projected/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-kube-api-access-764kb\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.791095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.791229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-wtmp\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:46:28.791318    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:46:28.791372    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls podName:d010c36a-0e87-479f-b4ee-ab12a3ecf6f9 nodeName:}" failed. No retries permitted until 2026-07-30 03:46:29.291353773 +0000 UTC m=+82.108290241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls") pod "node-exporter-99blf" (UID: "d010c36a-0e87-479f-b4ee-ab12a3ecf6f9") : secret "node-exporter-tls" not found
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.791989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-metrics-client-ca\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.792050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-sys\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.792090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-root\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.800092 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.792668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-textfile\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.803683 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.803656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:28.809127 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:28.809081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764kb\" (UniqueName: \"kubernetes.io/projected/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-kube-api-access-764kb\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:29.294752 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:29.294720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:29.297175 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:29.297154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d010c36a-0e87-479f-b4ee-ab12a3ecf6f9-node-exporter-tls\") pod \"node-exporter-99blf\" (UID: \"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9\") " pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:29.488274 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:29.488240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99blf"
Jul 30 03:46:29.900512 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:29.900384    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:46:30.253214 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:30.253183    2562 patch_prober.go:28] interesting pod/image-registry-f7b4dff4f-ddcfr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 03:46:30.253386 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:30.253234    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr" podUID="acf349c0-f872-405e-b222-a80397d52f0f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:46:30.692114 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:46:30.692089    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd010c36a_0e87_479f_b4ee_ab12a3ecf6f9.slice/crio-67987eae7900d61b549fc3799d5d60b68630569df21a17c0238c1b9278216aee WatchSource:0}: Error finding container 67987eae7900d61b549fc3799d5d60b68630569df21a17c0238c1b9278216aee: Status 404 returned error can't find the container with id 67987eae7900d61b549fc3799d5d60b68630569df21a17c0238c1b9278216aee
Jul 30 03:46:31.263959 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:31.263918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99blf" event={"ID":"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9","Type":"ContainerStarted","Data":"67987eae7900d61b549fc3799d5d60b68630569df21a17c0238c1b9278216aee"}
Jul 30 03:46:31.266352 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:31.266322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v2jpw" event={"ID":"0e839a7c-89b6-4915-a518-16ecbd17e771","Type":"ContainerStarted","Data":"445c612d6b7da4b2e67419395c3933582d59f64e516ccd742ceb342b04b783f4"}
Jul 30 03:46:31.284944 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:31.284904    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v2jpw" podStartSLOduration=33.483257453 podStartE2EDuration="45.284890675s" totalImagesPullingTime="11.801633222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 03:45:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:31.283908635 +0000 UTC m=+84.100845123" watchObservedRunningTime="2026-07-30 03:46:31.284890675 +0000 UTC m=+84.101827161"
Jul 30 03:46:32.151440 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:32.151409    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f7b4dff4f-ddcfr"
Jul 30 03:46:32.270955 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:32.270921    2562 generic.go:356] "Generic (PLEG): container finished" podID="d010c36a-0e87-479f-b4ee-ab12a3ecf6f9" containerID="52019c68b55b8bf1feca57fb10aa31fc86fee87c04d5c28cf6145311d81c2b60" exitCode=0
Jul 30 03:46:32.271110 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:32.271002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99blf" event={"ID":"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9","Type":"ContainerDied","Data":"52019c68b55b8bf1feca57fb10aa31fc86fee87c04d5c28cf6145311d81c2b60"}
Jul 30 03:46:33.278384 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:33.278348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99blf" event={"ID":"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9","Type":"ContainerStarted","Data":"e800ef6aa85466ab80f3ba95f04743867934fd93e932fe1f2c2d7e63a417b5d8"}
Jul 30 03:46:33.278881 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:33.278453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99blf" event={"ID":"d010c36a-0e87-479f-b4ee-ab12a3ecf6f9","Type":"ContainerStarted","Data":"79c2abb56563382a7ca5fbe72908af727e9285ebc4c0f25b4054bc3a1db9008d"}
Jul 30 03:46:33.302997 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:33.302940    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-99blf" podStartSLOduration=4.347398549 podStartE2EDuration="5.302921782s" totalImagesPullingTime="895.20905ms" totalInitContainerRuntime="60.314183ms" isStatefulPod=true podCreationTimestamp="2026-07-30 03:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:46:33.3014589 +0000 UTC m=+86.118395396" watchObservedRunningTime="2026-07-30 03:46:33.302921782 +0000 UTC m=+86.119858257"
Jul 30 03:46:34.913547 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:34.913478    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerName="registry" containerID="cri-o://277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654" gracePeriod=30
Jul 30 03:46:35.148266 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.148242    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:46:35.287779 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.287750    2562 generic.go:356] "Generic (PLEG): container finished" podID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerID="277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654" exitCode=0
Jul 30 03:46:35.287920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.287793    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5"
Jul 30 03:46:35.287920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.287837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" event={"ID":"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e","Type":"ContainerDied","Data":"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"}
Jul 30 03:46:35.287920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.287887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74946bc9c6-kmqw5" event={"ID":"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e","Type":"ContainerDied","Data":"9181e0b5998080328a908a4d55d2a2379c22e1e1c255fbce63836af77e0314c5"}
Jul 30 03:46:35.287920 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.287907    2562 scope.go:117] "RemoveContainer" containerID="277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"
Jul 30 03:46:35.295476 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.295456    2562 scope.go:117] "RemoveContainer" containerID="277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"
Jul 30 03:46:35.295741 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:46:35.295724    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654\": container with ID starting with 277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654 not found: ID does not exist" containerID="277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"
Jul 30 03:46:35.295795 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.295747    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654"} err="failed to get container status \"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654\": rpc error: code = NotFound desc = could not find container \"277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654\": container with ID starting with 277c69d04d07b60d82c1cf86bbbbd5e78bf34f081ee2a1f3c7845bbb1c78c654 not found: ID does not exist"
Jul 30 03:46:35.341008 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.340985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341103 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341024    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttz8j\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341103 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341203 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341102    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341203 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341134    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341203 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341175    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341203    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates\") pod \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\" (UID: \"be0bef51-e4c5-4cc4-a477-12ddba6ffe6e\") "
Jul 30 03:46:35.341811 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:46:35.341811 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.341754    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:46:35.343591 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.343551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:46:35.343687 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.343583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:46:35.343687 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.343677    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:46:35.343798 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.343692    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j" (OuterVolumeSpecName: "kube-api-access-ttz8j") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "kube-api-access-ttz8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:46:35.343798 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.343742    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:46:35.349330 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.349307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" (UID: "be0bef51-e4c5-4cc4-a477-12ddba6ffe6e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:46:35.442550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442511    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-ca-trust-extracted\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442547    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-installation-pull-secrets\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442557    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-image-registry-private-configuration\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442566    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-trusted-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442575    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-bound-sa-token\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442583    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-certificates\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442591    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-registry-tls\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.442654 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.442598    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttz8j\" (UniqueName: \"kubernetes.io/projected/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e-kube-api-access-ttz8j\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:46:35.610433 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.610407    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:46:35.613882 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.613863    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74946bc9c6-kmqw5"]
Jul 30 03:46:35.783966 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:35.783929    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" path="/var/lib/kubelet/pods/be0bef51-e4c5-4cc4-a477-12ddba6ffe6e/volumes"
Jul 30 03:46:47.204936 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:47.204903    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s5n46"
Jul 30 03:46:58.372561 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.372511    2562 generic.go:356] "Generic (PLEG): container finished" podID="92912f38-2f66-4711-90e3-13fa6dada35c" containerID="1f7eab1cbbc47cd5b5ff18486be9eae16051206edce79c5b6b39996964a430eb" exitCode=0
Jul 30 03:46:58.372985 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.372604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785" event={"ID":"92912f38-2f66-4711-90e3-13fa6dada35c","Type":"ContainerDied","Data":"1f7eab1cbbc47cd5b5ff18486be9eae16051206edce79c5b6b39996964a430eb"}
Jul 30 03:46:58.373061 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.373002    2562 scope.go:117] "RemoveContainer" containerID="1f7eab1cbbc47cd5b5ff18486be9eae16051206edce79c5b6b39996964a430eb"
Jul 30 03:46:58.374066 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.374046    2562 generic.go:356] "Generic (PLEG): container finished" podID="55773a70-3b09-4801-82a8-e1dee6bdf633" containerID="473057e0521335727313f2fe3ad919113d1fb9fe205729d123c3dbdc3f1ca13c" exitCode=0
Jul 30 03:46:58.374144 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.374078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t" event={"ID":"55773a70-3b09-4801-82a8-e1dee6bdf633","Type":"ContainerDied","Data":"473057e0521335727313f2fe3ad919113d1fb9fe205729d123c3dbdc3f1ca13c"}
Jul 30 03:46:58.374356 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:58.374343    2562 scope.go:117] "RemoveContainer" containerID="473057e0521335727313f2fe3ad919113d1fb9fe205729d123c3dbdc3f1ca13c"
Jul 30 03:46:59.379375 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:59.379339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-tq785" event={"ID":"92912f38-2f66-4711-90e3-13fa6dada35c","Type":"ContainerStarted","Data":"f845af3a91a0056fe43a28539bd0c18d6398faa3b451095e1603dfd4c56b1f24"}
Jul 30 03:46:59.380960 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:46:59.380940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t" event={"ID":"55773a70-3b09-4801-82a8-e1dee6bdf633","Type":"ContainerStarted","Data":"d8bfdf541808808ce7c09c49ff7faeecaf242f183c7fd5a6984b4e48d1ac3777"}
Jul 30 03:47:10.314407 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:47:10.314374    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/init-textfile/0.log"
Jul 30 03:47:10.514742 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:47:10.514713    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/node-exporter/0.log"
Jul 30 03:47:10.714267 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:47:10.714200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/kube-rbac-proxy/0.log"
Jul 30 03:47:17.713690 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:47:17.713662    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b9dc4d85-dz8bz_d1347232-d5a8-4c77-b0de-2609a00e9b05/router/0.log"
Jul 30 03:47:18.113984 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:47:18.113956    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6fhsw_76f692da-ea9a-4623-8ee6-20a20440f4d7/serve-healthcheck-canary/0.log"
Jul 30 03:49:19.825130 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.825085    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"]
Jul 30 03:49:19.825675 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.825382    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerName="registry"
Jul 30 03:49:19.825675 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.825395    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerName="registry"
Jul 30 03:49:19.825675 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.825450    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0bef51-e4c5-4cc4-a477-12ddba6ffe6e" containerName="registry"
Jul 30 03:49:19.828599 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.828571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:19.833843 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.833807    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 03:49:19.834006 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.833964    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 03:49:19.834006 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.833987    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mj4sc\""
Jul 30 03:49:19.834006 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.833995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 03:49:19.834171 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.834146    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 03:49:19.843796 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.843770    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"]
Jul 30 03:49:19.951902 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.951877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfk5\" (UniqueName: \"kubernetes.io/projected/e1031033-e23b-4455-a957-7e449137dfc2-kube-api-access-xsfk5\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:19.952019 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.951907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:19.952019 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.951941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:19.952019 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.951958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:19.952019 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:19.951979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfk5\" (UniqueName: \"kubernetes.io/projected/e1031033-e23b-4455-a957-7e449137dfc2-kube-api-access-xsfk5\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053310 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.053704 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.053674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-manifests\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.055632 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.055612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1031033-e23b-4455-a957-7e449137dfc2-tmp\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.055888 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.055871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.056039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.056018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1031033-e23b-4455-a957-7e449137dfc2-webhook-cert\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.061508 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.061484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfk5\" (UniqueName: \"kubernetes.io/projected/e1031033-e23b-4455-a957-7e449137dfc2-kube-api-access-xsfk5\") pod \"opendatahub-operator-controller-manager-599789d45f-n9nv5\" (UID: \"e1031033-e23b-4455-a957-7e449137dfc2\") " pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.139703 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.139646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:20.273768 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.273743    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"]
Jul 30 03:49:20.276172 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:20.276144    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1031033_e23b_4455_a957_7e449137dfc2.slice/crio-08589fa6f8e8731fcdef06bc4f82edfdd6bb3fb1c430475d559f56f62a79a90a WatchSource:0}: Error finding container 08589fa6f8e8731fcdef06bc4f82edfdd6bb3fb1c430475d559f56f62a79a90a: Status 404 returned error can't find the container with id 08589fa6f8e8731fcdef06bc4f82edfdd6bb3fb1c430475d559f56f62a79a90a
Jul 30 03:49:20.882251 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:20.882207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5" event={"ID":"e1031033-e23b-4455-a957-7e449137dfc2","Type":"ContainerStarted","Data":"08589fa6f8e8731fcdef06bc4f82edfdd6bb3fb1c430475d559f56f62a79a90a"}
Jul 30 03:49:23.896705 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:23.896671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5" event={"ID":"e1031033-e23b-4455-a957-7e449137dfc2","Type":"ContainerStarted","Data":"6e869cdddcf7fbcc5b975c5c6d02feacd9296756251e6aaf75fb59962b3ce495"}
Jul 30 03:49:24.904596 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:24.904558    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1031033-e23b-4455-a957-7e449137dfc2" containerID="6e869cdddcf7fbcc5b975c5c6d02feacd9296756251e6aaf75fb59962b3ce495" exitCode=0
Jul 30 03:49:24.905081 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:24.904645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5" event={"ID":"e1031033-e23b-4455-a957-7e449137dfc2","Type":"ContainerDied","Data":"6e869cdddcf7fbcc5b975c5c6d02feacd9296756251e6aaf75fb59962b3ce495"}
Jul 30 03:49:25.911123 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:25.911080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5" event={"ID":"e1031033-e23b-4455-a957-7e449137dfc2","Type":"ContainerStarted","Data":"4423fb1f6db6819c216eff85f65eac8fb1f0ac2b86846bcc77d51edd73195556"}
Jul 30 03:49:25.911626 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:25.911213    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:25.932262 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:25.932202    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5" podStartSLOduration=2.772849182 podStartE2EDuration="6.932187949s" totalImagesPullingTime="3.1342316s" totalInitContainerRuntime="1.025107167s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:25.930290877 +0000 UTC m=+258.747227349" watchObservedRunningTime="2026-07-30 03:49:25.932187949 +0000 UTC m=+258.749124425"
Jul 30 03:49:36.917015 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:36.916937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-599789d45f-n9nv5"
Jul 30 03:49:40.176641 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.176606    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"]
Jul 30 03:49:40.183198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.183173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.186270 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.186253    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 30 03:49:40.186388 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.186365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 30 03:49:40.187229 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.187209    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qhd9l\""
Jul 30 03:49:40.191847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.191821    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"]
Jul 30 03:49:40.294839 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.294803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10fd0823-093e-4bfc-97ac-a14dad793f8c-tmp\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.294839 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.294838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fd0823-093e-4bfc-97ac-a14dad793f8c-tls-certs\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.295040 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.294872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9lz\" (UniqueName: \"kubernetes.io/projected/10fd0823-093e-4bfc-97ac-a14dad793f8c-kube-api-access-8g9lz\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.335478 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.335446    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jc2b7"]
Jul 30 03:49:40.337954 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.337939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.340941 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.340923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 30 03:49:40.342243 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.342228    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2t4k4\""
Jul 30 03:49:40.353361 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.353335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jc2b7"]
Jul 30 03:49:40.395494 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.395468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6m48\" (UniqueName: \"kubernetes.io/projected/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-kube-api-access-t6m48\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.395623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.395500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-cert\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.395623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.395602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10fd0823-093e-4bfc-97ac-a14dad793f8c-tmp\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.395729 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.395630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fd0823-093e-4bfc-97ac-a14dad793f8c-tls-certs\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.395729 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.395657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9lz\" (UniqueName: \"kubernetes.io/projected/10fd0823-093e-4bfc-97ac-a14dad793f8c-kube-api-access-8g9lz\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.397818 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.397798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10fd0823-093e-4bfc-97ac-a14dad793f8c-tmp\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.397977 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.397961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fd0823-093e-4bfc-97ac-a14dad793f8c-tls-certs\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.403947 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.403925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9lz\" (UniqueName: \"kubernetes.io/projected/10fd0823-093e-4bfc-97ac-a14dad793f8c-kube-api-access-8g9lz\") pod \"kube-auth-proxy-58ccc58984-n82rr\" (UID: \"10fd0823-093e-4bfc-97ac-a14dad793f8c\") " pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.492619 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.492598    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"
Jul 30 03:49:40.496746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.496722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6m48\" (UniqueName: \"kubernetes.io/projected/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-kube-api-access-t6m48\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.496843 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.496832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-cert\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.499108 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.499089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-cert\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.508470 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.508442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6m48\" (UniqueName: \"kubernetes.io/projected/81fc883b-e8d5-4b2b-b791-5a5d6daf066f-kube-api-access-t6m48\") pod \"odh-model-controller-7998c67dc4-jc2b7\" (UID: \"81fc883b-e8d5-4b2b-b791-5a5d6daf066f\") " pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.625311 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.622724    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58ccc58984-n82rr"]
Jul 30 03:49:40.647633 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.647608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:40.772673 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.772602    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jc2b7"]
Jul 30 03:49:40.775693 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:40.775668    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fc883b_e8d5_4b2b_b791_5a5d6daf066f.slice/crio-64aca09a26691f5677ca7da8443cd486330a28af9b19be4e7e9bdbc8811fcb4b WatchSource:0}: Error finding container 64aca09a26691f5677ca7da8443cd486330a28af9b19be4e7e9bdbc8811fcb4b: Status 404 returned error can't find the container with id 64aca09a26691f5677ca7da8443cd486330a28af9b19be4e7e9bdbc8811fcb4b
Jul 30 03:49:40.943750 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.943719    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2hlrv"]
Jul 30 03:49:40.947461 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.947443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:40.949979 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.949961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-625bt\""
Jul 30 03:49:40.950065 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.949961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 30 03:49:40.956690 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.956658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2hlrv"]
Jul 30 03:49:40.967668 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.967643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr" event={"ID":"10fd0823-093e-4bfc-97ac-a14dad793f8c","Type":"ContainerStarted","Data":"755a9d6660c204117bb4b975fdb518a299c499f179ea832240224c026758c41a"}
Jul 30 03:49:40.968973 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:40.968949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7" event={"ID":"81fc883b-e8d5-4b2b-b791-5a5d6daf066f","Type":"ContainerStarted","Data":"64aca09a26691f5677ca7da8443cd486330a28af9b19be4e7e9bdbc8811fcb4b"}
Jul 30 03:49:41.102191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.102132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490420af-8cae-4a85-86e2-596af1e63beb-tls-certs\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.102191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.102176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6dq\" (UniqueName: \"kubernetes.io/projected/490420af-8cae-4a85-86e2-596af1e63beb-kube-api-access-rl6dq\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.203047 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.203022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490420af-8cae-4a85-86e2-596af1e63beb-tls-certs\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.203333 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.203061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6dq\" (UniqueName: \"kubernetes.io/projected/490420af-8cae-4a85-86e2-596af1e63beb-kube-api-access-rl6dq\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.205309 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.205284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490420af-8cae-4a85-86e2-596af1e63beb-tls-certs\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.214006 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.213987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6dq\" (UniqueName: \"kubernetes.io/projected/490420af-8cae-4a85-86e2-596af1e63beb-kube-api-access-rl6dq\") pod \"model-serving-api-54d89bd5fc-2hlrv\" (UID: \"490420af-8cae-4a85-86e2-596af1e63beb\") " pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.258919 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.258897    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:41.407160 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.406649    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2hlrv"]
Jul 30 03:49:41.415016 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:41.414961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490420af_8cae_4a85_86e2_596af1e63beb.slice/crio-81409cd3c8d0eb486668ce23d52d03830926a940ca384b50b6834b8f884b658c WatchSource:0}: Error finding container 81409cd3c8d0eb486668ce23d52d03830926a940ca384b50b6834b8f884b658c: Status 404 returned error can't find the container with id 81409cd3c8d0eb486668ce23d52d03830926a940ca384b50b6834b8f884b658c
Jul 30 03:49:41.429788 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.429762    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:41.433332 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.433312    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.435991 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.435942    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 30 03:49:41.436172 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.436135    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 30 03:49:41.436501 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.436463    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 30 03:49:41.436671 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.436649    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 30 03:49:41.436859 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.436840    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2s9s\""
Jul 30 03:49:41.445558 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.445535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:41.504897 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.504866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505030 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.504921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505030 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.504954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505139 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.505027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505139 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.505054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505139 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.505091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnx4l\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.505139 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.505115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx4l\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.605740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.607039 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.606968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.609480 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.609417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.610251 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.610210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.610418 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.610378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.611450 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.611429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.614323 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.614281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx4l\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.616446 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.616423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-hwfrb\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.746668 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.746388    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:41.922932 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.922857    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:41.924434 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:41.924404    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fc8105_b1a9_4c18_a4de_84b5f325576f.slice/crio-9cbc590ee985f06d651006117c60dd87615946aa855ef0e46d41257b54d88acb WatchSource:0}: Error finding container 9cbc590ee985f06d651006117c60dd87615946aa855ef0e46d41257b54d88acb: Status 404 returned error can't find the container with id 9cbc590ee985f06d651006117c60dd87615946aa855ef0e46d41257b54d88acb
Jul 30 03:49:41.976534 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.976493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv" event={"ID":"490420af-8cae-4a85-86e2-596af1e63beb","Type":"ContainerStarted","Data":"81409cd3c8d0eb486668ce23d52d03830926a940ca384b50b6834b8f884b658c"}
Jul 30 03:49:41.977981 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:41.977917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" event={"ID":"e6fc8105-b1a9-4c18-a4de-84b5f325576f","Type":"ContainerStarted","Data":"9cbc590ee985f06d651006117c60dd87615946aa855ef0e46d41257b54d88acb"}
Jul 30 03:49:45.838742 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:45.838703    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 03:49:45.839018 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:45.838778    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 03:49:45.998158 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:45.998126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr" event={"ID":"10fd0823-093e-4bfc-97ac-a14dad793f8c","Type":"ContainerStarted","Data":"33241ef8dc4857b3821ed07e2ed4ce5db2c768a74390e6ab03833f1ae9b8a552"}
Jul 30 03:49:45.999692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:45.999660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7" event={"ID":"81fc883b-e8d5-4b2b-b791-5a5d6daf066f","Type":"ContainerStarted","Data":"e7cfd925957c99c27f9179c672a442eac6cc6bf9da1f8dac3d4ea46cda1c46cd"}
Jul 30 03:49:45.999819 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:45.999712    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:46.001087 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.001054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" event={"ID":"e6fc8105-b1a9-4c18-a4de-84b5f325576f","Type":"ContainerStarted","Data":"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"}
Jul 30 03:49:46.001231 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.001185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:46.002513 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.002489    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-hwfrb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:49:46.002642 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.002553    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:49:46.002642 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.002586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv" event={"ID":"490420af-8cae-4a85-86e2-596af1e63beb","Type":"ContainerStarted","Data":"307bba3cd3edc635aedfe08a8942b2d223230fa96ae91d73bfa7c1560f1240ae"}
Jul 30 03:49:46.002772 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.002756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:49:46.013946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.013907    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58ccc58984-n82rr" podStartSLOduration=0.87623018 podStartE2EDuration="6.013895644s" totalImagesPullingTime="5.137665464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:46.012959611 +0000 UTC m=+278.829896085" watchObservedRunningTime="2026-07-30 03:49:46.013895644 +0000 UTC m=+278.830832120"
Jul 30 03:49:46.030666 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.030573    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" podStartSLOduration=1.119328811 podStartE2EDuration="5.030562238s" totalImagesPullingTime="3.911233427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:46.029726907 +0000 UTC m=+278.846663426" watchObservedRunningTime="2026-07-30 03:49:46.030562238 +0000 UTC m=+278.847498692"
Jul 30 03:49:46.045418 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:46.045370    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7" podStartSLOduration=1.057065803 podStartE2EDuration="6.045353673s" totalImagesPullingTime="4.98828787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:46.045030241 +0000 UTC m=+278.861966721" watchObservedRunningTime="2026-07-30 03:49:46.045353673 +0000 UTC m=+278.862290149"
Jul 30 03:49:47.007889 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:47.007847    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-hwfrb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 03:49:47.008319 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:47.007906    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 03:49:50.008219 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.008177    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:50.035805 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.035743    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv" podStartSLOduration=5.688553971 podStartE2EDuration="10.035721669s" totalImagesPullingTime="4.347167698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:46.061699519 +0000 UTC m=+278.878635995" watchObservedRunningTime="2026-07-30 03:49:50.035721669 +0000 UTC m=+282.852658146"
Jul 30 03:49:50.241684 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.241654    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"]
Jul 30 03:49:50.244927 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.244911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.248345 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.248319    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 30 03:49:50.248892 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.248873    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4v5hb\""
Jul 30 03:49:50.251264 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.251243    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"]
Jul 30 03:49:50.254791 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.254769    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.257943 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.257923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7fhzm\""
Jul 30 03:49:50.258153 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.258132    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 30 03:49:50.263915 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.263894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"]
Jul 30 03:49:50.271923 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.271902    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"]
Jul 30 03:49:50.381980 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.381939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a793978b-1b44-4de8-950c-9a2d759a5a06-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.382174 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.382002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.382174 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.382079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5nqg\" (UniqueName: \"kubernetes.io/projected/03376731-166b-441e-91c4-c5d2e3152555-kube-api-access-v5nqg\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.382174 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.382133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tjp\" (UniqueName: \"kubernetes.io/projected/a793978b-1b44-4de8-950c-9a2d759a5a06-kube-api-access-w8tjp\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.482474 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.482440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tjp\" (UniqueName: \"kubernetes.io/projected/a793978b-1b44-4de8-950c-9a2d759a5a06-kube-api-access-w8tjp\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.482657 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.482501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a793978b-1b44-4de8-950c-9a2d759a5a06-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.482657 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.482559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.482657 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.482587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5nqg\" (UniqueName: \"kubernetes.io/projected/03376731-166b-441e-91c4-c5d2e3152555-kube-api-access-v5nqg\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.482812 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:50.482689    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 30 03:49:50.482812 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:50.482755    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert podName:03376731-166b-441e-91c4-c5d2e3152555 nodeName:}" failed. No retries permitted until 2026-07-30 03:49:50.982735068 +0000 UTC m=+283.799671538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert") pod "kserve-controller-manager-d6f4b65b8-bdxxf" (UID: "03376731-166b-441e-91c4-c5d2e3152555") : secret "kserve-webhook-server-cert" not found
Jul 30 03:49:50.484772 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.484754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a793978b-1b44-4de8-950c-9a2d759a5a06-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.496689 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.496657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5nqg\" (UniqueName: \"kubernetes.io/projected/03376731-166b-441e-91c4-c5d2e3152555-kube-api-access-v5nqg\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.497054 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.497033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tjp\" (UniqueName: \"kubernetes.io/projected/a793978b-1b44-4de8-950c-9a2d759a5a06-kube-api-access-w8tjp\") pod \"llmisvc-controller-manager-777c6bd7d5-52dzh\" (UID: \"a793978b-1b44-4de8-950c-9a2d759a5a06\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.566108 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.566049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:50.698779 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.698749    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"]
Jul 30 03:49:50.701281 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:50.701253    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda793978b_1b44_4de8_950c_9a2d759a5a06.slice/crio-b1c77da53395ad83984c9bf32336a8f4bd237608218f2a6c3ce860b8332b601e WatchSource:0}: Error finding container b1c77da53395ad83984c9bf32336a8f4bd237608218f2a6c3ce860b8332b601e: Status 404 returned error can't find the container with id b1c77da53395ad83984c9bf32336a8f4bd237608218f2a6c3ce860b8332b601e
Jul 30 03:49:50.987239 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:50.987211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:50.987401 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:50.987293    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 30 03:49:50.987401 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:50.987333    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert podName:03376731-166b-441e-91c4-c5d2e3152555 nodeName:}" failed. No retries permitted until 2026-07-30 03:49:51.987321781 +0000 UTC m=+284.804258235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert") pod "kserve-controller-manager-d6f4b65b8-bdxxf" (UID: "03376731-166b-441e-91c4-c5d2e3152555") : secret "kserve-webhook-server-cert" not found
Jul 30 03:49:51.025750 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:51.025718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh" event={"ID":"a793978b-1b44-4de8-950c-9a2d759a5a06","Type":"ContainerStarted","Data":"b1c77da53395ad83984c9bf32336a8f4bd237608218f2a6c3ce860b8332b601e"}
Jul 30 03:49:51.995971 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:51.995926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:51.998715 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:51.998692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03376731-166b-441e-91c4-c5d2e3152555-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdxxf\" (UID: \"03376731-166b-441e-91c4-c5d2e3152555\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:52.057347 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:52.057319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:52.395367 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:52.395338    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"]
Jul 30 03:49:52.397240 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:49:52.397215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03376731_166b_441e_91c4_c5d2e3152555.slice/crio-f1ce5d227b2bf9287e27f584d36cddb8e928b528607936eab3fb7845ee9b2a84 WatchSource:0}: Error finding container f1ce5d227b2bf9287e27f584d36cddb8e928b528607936eab3fb7845ee9b2a84: Status 404 returned error can't find the container with id f1ce5d227b2bf9287e27f584d36cddb8e928b528607936eab3fb7845ee9b2a84
Jul 30 03:49:53.037768 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.037725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf" event={"ID":"03376731-166b-441e-91c4-c5d2e3152555","Type":"ContainerStarted","Data":"f1ce5d227b2bf9287e27f584d36cddb8e928b528607936eab3fb7845ee9b2a84"}
Jul 30 03:49:53.112487 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.112446    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:53.112925 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.112655    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery" containerID="cri-o://549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39" gracePeriod=30
Jul 30 03:49:53.415719 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.415697    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:53.510668 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510647    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510784 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510784 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510784 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510784 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510947 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.510947 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.510850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnx4l\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l\") pod \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\" (UID: \"e6fc8105-b1a9-4c18-a4de-84b5f325576f\") "
Jul 30 03:49:53.511044 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.511005    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 03:49:53.511160 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.511132    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-ca-configmap\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.512893 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.512858    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs" (OuterVolumeSpecName: "local-certs") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:49:53.513000 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.512943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts" (OuterVolumeSpecName: "cacerts") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:49:53.513155 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.513136    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:49:53.513246 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.513231    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:49:53.513326 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.513298    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:49:53.513423 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.513329    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l" (OuterVolumeSpecName: "kube-api-access-qnx4l") pod "e6fc8105-b1a9-4c18-a4de-84b5f325576f" (UID: "e6fc8105-b1a9-4c18-a4de-84b5f325576f"). InnerVolumeSpecName "kube-api-access-qnx4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:49:53.612313 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612254    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-token\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.612313 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612279    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnx4l\" (UniqueName: \"kubernetes.io/projected/e6fc8105-b1a9-4c18-a4de-84b5f325576f-kube-api-access-qnx4l\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.612313 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612294    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-csr-dns-cert\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.612313 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612306    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8105-b1a9-4c18-a4de-84b5f325576f-local-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.612313 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612318    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-istio-kubeconfig\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.612632 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:53.612325    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6fc8105-b1a9-4c18-a4de-84b5f325576f-cacerts\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:49:53.820580 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:53.820548    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fc8105_b1a9_4c18_a4de_84b5f325576f.slice\": RecentStats: unable to find data in memory cache]"
Jul 30 03:49:54.042628 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.042599    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerID="549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39" exitCode=0
Jul 30 03:49:54.042847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.042648    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"
Jul 30 03:49:54.042847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.042681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" event={"ID":"e6fc8105-b1a9-4c18-a4de-84b5f325576f","Type":"ContainerDied","Data":"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"}
Jul 30 03:49:54.042847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.042733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb" event={"ID":"e6fc8105-b1a9-4c18-a4de-84b5f325576f","Type":"ContainerDied","Data":"9cbc590ee985f06d651006117c60dd87615946aa855ef0e46d41257b54d88acb"}
Jul 30 03:49:54.042847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.042760    2562 scope.go:117] "RemoveContainer" containerID="549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"
Jul 30 03:49:54.044389 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.044363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh" event={"ID":"a793978b-1b44-4de8-950c-9a2d759a5a06","Type":"ContainerStarted","Data":"70bab7e6171802cd93fb8ea6d53fe18f171d8431a768fa1386e2f9e7cb228d91"}
Jul 30 03:49:54.044502 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.044415    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:49:54.052743 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.052721    2562 scope.go:117] "RemoveContainer" containerID="549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"
Jul 30 03:49:54.053481 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:49:54.053272    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39\": container with ID starting with 549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39 not found: ID does not exist" containerID="549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"
Jul 30 03:49:54.053596 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.053486    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39"} err="failed to get container status \"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39\": rpc error: code = NotFound desc = could not find container \"549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39\": container with ID starting with 549aed6ccf63648731ab3ec653d4010fcad173d875bf5a3a8b1c98ff701eba39 not found: ID does not exist"
Jul 30 03:49:54.068075 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.068057    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:54.071718 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.071698    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-hwfrb"]
Jul 30 03:49:54.085430 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:54.085385    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh" podStartSLOduration=1.4508830129999999 podStartE2EDuration="4.085370177s" totalImagesPullingTime="2.634487164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:54.084599991 +0000 UTC m=+286.901536466" watchObservedRunningTime="2026-07-30 03:49:54.085370177 +0000 UTC m=+286.902306653"
Jul 30 03:49:55.784918 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:55.784885    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" path="/var/lib/kubelet/pods/e6fc8105-b1a9-4c18-a4de-84b5f325576f/volumes"
Jul 30 03:49:56.055824 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:56.055738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf" event={"ID":"03376731-166b-441e-91c4-c5d2e3152555","Type":"ContainerStarted","Data":"6b2638b75895d2609635d37bb460bf4d98e12006fae5624ea3bb505552bf471e"}
Jul 30 03:49:56.055824 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:56.055802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:49:56.072535 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:56.072472    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf" podStartSLOduration=3.407956333 podStartE2EDuration="6.072459115s" totalImagesPullingTime="2.664502782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:49:56.071428722 +0000 UTC m=+288.888365200" watchObservedRunningTime="2026-07-30 03:49:56.072459115 +0000 UTC m=+288.889395590"
Jul 30 03:49:57.009773 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:57.009744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jc2b7"
Jul 30 03:49:57.011947 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:49:57.011918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2hlrv"
Jul 30 03:50:07.644973 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:07.644941    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 03:50:25.052199 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:25.052168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-52dzh"
Jul 30 03:50:27.066024 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:27.065997    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdxxf"
Jul 30 03:50:41.214812 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.214776    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:41.215173 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.215103    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery"
Jul 30 03:50:41.215173 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.215116    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery"
Jul 30 03:50:41.215242 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.215188    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fc8105-b1a9-4c18-a4de-84b5f325576f" containerName="discovery"
Jul 30 03:50:41.218186 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.218163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:41.224406 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.221371    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-swmvp\""
Jul 30 03:50:41.224406 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.221718    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 03:50:41.224406 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.222381    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 03:50:41.227038 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.227011    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:41.264760 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.264733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhf4\" (UniqueName: \"kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4\") pod \"kuadrant-operator-catalog-hzbqb\" (UID: \"199f4bfe-a3c0-441d-ab84-71547d24e89d\") " pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:41.365528 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.365503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhf4\" (UniqueName: \"kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4\") pod \"kuadrant-operator-catalog-hzbqb\" (UID: \"199f4bfe-a3c0-441d-ab84-71547d24e89d\") " pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:41.373357 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.373331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhf4\" (UniqueName: \"kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4\") pod \"kuadrant-operator-catalog-hzbqb\" (UID: \"199f4bfe-a3c0-441d-ab84-71547d24e89d\") " pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:41.535385 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.535360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:41.600776 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.600738    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:41.655450 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.655390    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:41.657656 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:50:41.657628    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199f4bfe_a3c0_441d_ab84_71547d24e89d.slice/crio-620081eff785857c99d2518311a1d84bafe7ae2e8924a8afb08537c24e1efda3 WatchSource:0}: Error finding container 620081eff785857c99d2518311a1d84bafe7ae2e8924a8afb08537c24e1efda3: Status 404 returned error can't find the container with id 620081eff785857c99d2518311a1d84bafe7ae2e8924a8afb08537c24e1efda3
Jul 30 03:50:41.659205 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.659183    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:50:41.798101 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.798037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8gjnc"]
Jul 30 03:50:41.802307 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.802291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:41.808698 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.808667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8gjnc"]
Jul 30 03:50:41.870067 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.870045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577qg\" (UniqueName: \"kubernetes.io/projected/cd8774ac-a070-4410-af23-45d6722f4f92-kube-api-access-577qg\") pod \"kuadrant-operator-catalog-8gjnc\" (UID: \"cd8774ac-a070-4410-af23-45d6722f4f92\") " pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:41.970999 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.970973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577qg\" (UniqueName: \"kubernetes.io/projected/cd8774ac-a070-4410-af23-45d6722f4f92-kube-api-access-577qg\") pod \"kuadrant-operator-catalog-8gjnc\" (UID: \"cd8774ac-a070-4410-af23-45d6722f4f92\") " pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:41.978084 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:41.978064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-577qg\" (UniqueName: \"kubernetes.io/projected/cd8774ac-a070-4410-af23-45d6722f4f92-kube-api-access-577qg\") pod \"kuadrant-operator-catalog-8gjnc\" (UID: \"cd8774ac-a070-4410-af23-45d6722f4f92\") " pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:42.112675 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:42.112611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:42.234739 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:42.234713    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8gjnc"]
Jul 30 03:50:42.236852 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:50:42.236820    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8774ac_a070_4410_af23_45d6722f4f92.slice/crio-ccd11b760102d2730aedb6f9abed2bcf27d42a68fbc1a9f48519674bc965f66f WatchSource:0}: Error finding container ccd11b760102d2730aedb6f9abed2bcf27d42a68fbc1a9f48519674bc965f66f: Status 404 returned error can't find the container with id ccd11b760102d2730aedb6f9abed2bcf27d42a68fbc1a9f48519674bc965f66f
Jul 30 03:50:42.241485 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:42.241457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" event={"ID":"199f4bfe-a3c0-441d-ab84-71547d24e89d","Type":"ContainerStarted","Data":"620081eff785857c99d2518311a1d84bafe7ae2e8924a8afb08537c24e1efda3"}
Jul 30 03:50:43.246857 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:43.246822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc" event={"ID":"cd8774ac-a070-4410-af23-45d6722f4f92","Type":"ContainerStarted","Data":"ccd11b760102d2730aedb6f9abed2bcf27d42a68fbc1a9f48519674bc965f66f"}
Jul 30 03:50:44.252925 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.252888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" event={"ID":"199f4bfe-a3c0-441d-ab84-71547d24e89d","Type":"ContainerStarted","Data":"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"}
Jul 30 03:50:44.252925 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.252905    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" podUID="199f4bfe-a3c0-441d-ab84-71547d24e89d" containerName="registry-server" containerID="cri-o://3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7" gracePeriod=2
Jul 30 03:50:44.254383 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.254359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc" event={"ID":"cd8774ac-a070-4410-af23-45d6722f4f92","Type":"ContainerStarted","Data":"6d2963dbfc3e6540beb3d2b83961a5e18fd9507ed660970649de251ea2ee47a8"}
Jul 30 03:50:44.267236 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.267199    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" podStartSLOduration=1.241323962 podStartE2EDuration="3.267186688s" totalImagesPullingTime="2.025862726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:50:44.266987226 +0000 UTC m=+337.083923705" watchObservedRunningTime="2026-07-30 03:50:44.267186688 +0000 UTC m=+337.084123172"
Jul 30 03:50:44.281643 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.281599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc" podStartSLOduration=1.833861899 podStartE2EDuration="3.281579296s" totalImagesPullingTime="1.447717397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:50:44.279897164 +0000 UTC m=+337.096833643" watchObservedRunningTime="2026-07-30 03:50:44.281579296 +0000 UTC m=+337.098515774"
Jul 30 03:50:44.493275 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.493249    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:44.594653 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.594561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjhf4\" (UniqueName: \"kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4\") pod \"199f4bfe-a3c0-441d-ab84-71547d24e89d\" (UID: \"199f4bfe-a3c0-441d-ab84-71547d24e89d\") "
Jul 30 03:50:44.596646 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.596622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4" (OuterVolumeSpecName: "kube-api-access-fjhf4") pod "199f4bfe-a3c0-441d-ab84-71547d24e89d" (UID: "199f4bfe-a3c0-441d-ab84-71547d24e89d"). InnerVolumeSpecName "kube-api-access-fjhf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:50:44.695685 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:44.695651    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhf4\" (UniqueName: \"kubernetes.io/projected/199f4bfe-a3c0-441d-ab84-71547d24e89d-kube-api-access-fjhf4\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:50:45.259365 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.259334    2562 generic.go:356] "Generic (PLEG): container finished" podID="199f4bfe-a3c0-441d-ab84-71547d24e89d" containerID="3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7" exitCode=0
Jul 30 03:50:45.259830 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.259379    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb"
Jul 30 03:50:45.259830 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.259421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" event={"ID":"199f4bfe-a3c0-441d-ab84-71547d24e89d","Type":"ContainerDied","Data":"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"}
Jul 30 03:50:45.259830 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.259478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzbqb" event={"ID":"199f4bfe-a3c0-441d-ab84-71547d24e89d","Type":"ContainerDied","Data":"620081eff785857c99d2518311a1d84bafe7ae2e8924a8afb08537c24e1efda3"}
Jul 30 03:50:45.259830 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.259512    2562 scope.go:117] "RemoveContainer" containerID="3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"
Jul 30 03:50:45.272550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.272511    2562 scope.go:117] "RemoveContainer" containerID="3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"
Jul 30 03:50:45.272804 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:50:45.272785    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7\": container with ID starting with 3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7 not found: ID does not exist" containerID="3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"
Jul 30 03:50:45.272869 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.272816    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7"} err="failed to get container status \"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7\": rpc error: code = NotFound desc = could not find container \"3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7\": container with ID starting with 3399552dd3ff38788e40e9cf723d3b1581fabfea8adfa690cd28b507aa1026c7 not found: ID does not exist"
Jul 30 03:50:45.284437 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.284407    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:45.290747 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.290727    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzbqb"]
Jul 30 03:50:45.783937 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:45.783905    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199f4bfe-a3c0-441d-ab84-71547d24e89d" path="/var/lib/kubelet/pods/199f4bfe-a3c0-441d-ab84-71547d24e89d/volumes"
Jul 30 03:50:52.113186 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:52.113152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:52.113627 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:52.113226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:52.134089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:52.134061    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:50:52.314679 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:50:52.314652    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8gjnc"
Jul 30 03:51:12.532936 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.532905    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"]
Jul 30 03:51:12.533314 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.533208    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199f4bfe-a3c0-441d-ab84-71547d24e89d" containerName="registry-server"
Jul 30 03:51:12.533314 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.533219    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="199f4bfe-a3c0-441d-ab84-71547d24e89d" containerName="registry-server"
Jul 30 03:51:12.533314 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.533278    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="199f4bfe-a3c0-441d-ab84-71547d24e89d" containerName="registry-server"
Jul 30 03:51:12.541077 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.541062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:12.544249 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.544222    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fvsjt\""
Jul 30 03:51:12.544764 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.544747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 30 03:51:12.548114 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.548088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"]
Jul 30 03:51:12.719049 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.719010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkzw\" (UniqueName: \"kubernetes.io/projected/b843f183-8b45-4a1c-becb-4ea2dca5bdca-kube-api-access-trkzw\") pod \"dns-operator-controller-manager-864dd78c4c-rctsq\" (UID: \"b843f183-8b45-4a1c-becb-4ea2dca5bdca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:12.819968 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.819887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkzw\" (UniqueName: \"kubernetes.io/projected/b843f183-8b45-4a1c-becb-4ea2dca5bdca-kube-api-access-trkzw\") pod \"dns-operator-controller-manager-864dd78c4c-rctsq\" (UID: \"b843f183-8b45-4a1c-becb-4ea2dca5bdca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:12.828841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.828819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkzw\" (UniqueName: \"kubernetes.io/projected/b843f183-8b45-4a1c-becb-4ea2dca5bdca-kube-api-access-trkzw\") pod \"dns-operator-controller-manager-864dd78c4c-rctsq\" (UID: \"b843f183-8b45-4a1c-becb-4ea2dca5bdca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:12.851857 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.851829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:12.978453 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:12.978317    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"]
Jul 30 03:51:12.981371 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:12.981348    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb843f183_8b45_4a1c_becb_4ea2dca5bdca.slice/crio-a93e97a8320b3e16b36ddecd73739f9dbf60478d3dd64f5f5d23a0e0760e4b1f WatchSource:0}: Error finding container a93e97a8320b3e16b36ddecd73739f9dbf60478d3dd64f5f5d23a0e0760e4b1f: Status 404 returned error can't find the container with id a93e97a8320b3e16b36ddecd73739f9dbf60478d3dd64f5f5d23a0e0760e4b1f
Jul 30 03:51:13.393100 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:13.393067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq" event={"ID":"b843f183-8b45-4a1c-becb-4ea2dca5bdca","Type":"ContainerStarted","Data":"a93e97a8320b3e16b36ddecd73739f9dbf60478d3dd64f5f5d23a0e0760e4b1f"}
Jul 30 03:51:15.404386 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.404305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq" event={"ID":"b843f183-8b45-4a1c-becb-4ea2dca5bdca","Type":"ContainerStarted","Data":"b74e1eb55c203e631c916aa412f1e8d8730b3dc51f7675ca59ffea3102cc3f18"}
Jul 30 03:51:15.404774 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.404450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:15.421297 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.421253    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq" podStartSLOduration=1.4265623920000001 podStartE2EDuration="3.421239657s" totalImagesPullingTime="1.994677265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:15.419696712 +0000 UTC m=+368.236633191" watchObservedRunningTime="2026-07-30 03:51:15.421239657 +0000 UTC m=+368.238176186"
Jul 30 03:51:15.980912 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.980879    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"]
Jul 30 03:51:15.984296 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.984275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:15.986710 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.986695    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rlvqt\""
Jul 30 03:51:15.993216 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:15.993187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"]
Jul 30 03:51:16.163004 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:16.162969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57xk\" (UniqueName: \"kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk\") pod \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" (UID: \"e6c1a66c-e8fc-45ff-8bca-ac998916ace3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:16.264201 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:16.264120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57xk\" (UniqueName: \"kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk\") pod \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" (UID: \"e6c1a66c-e8fc-45ff-8bca-ac998916ace3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:16.277049 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:16.277021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57xk\" (UniqueName: \"kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk\") pod \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" (UID: \"e6c1a66c-e8fc-45ff-8bca-ac998916ace3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:16.293794 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:16.293770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:16.424392 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:16.424342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"]
Jul 30 03:51:16.426617 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:16.426591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c1a66c_e8fc_45ff_8bca_ac998916ace3.slice/crio-9dda090eb19f7daea34dd29d8d96f4a7aded2e08b80087c29082f52df2048cb7 WatchSource:0}: Error finding container 9dda090eb19f7daea34dd29d8d96f4a7aded2e08b80087c29082f52df2048cb7: Status 404 returned error can't find the container with id 9dda090eb19f7daea34dd29d8d96f4a7aded2e08b80087c29082f52df2048cb7
Jul 30 03:51:17.413847 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:17.413805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4" event={"ID":"e6c1a66c-e8fc-45ff-8bca-ac998916ace3","Type":"ContainerStarted","Data":"9dda090eb19f7daea34dd29d8d96f4a7aded2e08b80087c29082f52df2048cb7"}
Jul 30 03:51:18.419184 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:18.419146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4" event={"ID":"e6c1a66c-e8fc-45ff-8bca-ac998916ace3","Type":"ContainerStarted","Data":"2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f"}
Jul 30 03:51:18.419582 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:18.419215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:18.435881 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:18.435835    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4" podStartSLOduration=1.9166071850000002 podStartE2EDuration="3.43581995s" totalImagesPullingTime="1.519212765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:18.434458075 +0000 UTC m=+371.251394552" watchObservedRunningTime="2026-07-30 03:51:18.43581995 +0000 UTC m=+371.252756426"
Jul 30 03:51:19.881167 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.881133    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"]
Jul 30 03:51:19.884481 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.884465    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:19.887564 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.887549    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4xkk8\""
Jul 30 03:51:19.893616 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.893578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52f69\" (UniqueName: \"kubernetes.io/projected/7dc02ebc-a14d-4bb2-81a3-db3512a6a441-kube-api-access-52f69\") pod \"authorino-operator-75cc8d5cb7-xrjrb\" (UID: \"7dc02ebc-a14d-4bb2-81a3-db3512a6a441\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:19.895676 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.895655    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"]
Jul 30 03:51:19.994229 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:19.994190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52f69\" (UniqueName: \"kubernetes.io/projected/7dc02ebc-a14d-4bb2-81a3-db3512a6a441-kube-api-access-52f69\") pod \"authorino-operator-75cc8d5cb7-xrjrb\" (UID: \"7dc02ebc-a14d-4bb2-81a3-db3512a6a441\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:20.008946 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:20.008921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52f69\" (UniqueName: \"kubernetes.io/projected/7dc02ebc-a14d-4bb2-81a3-db3512a6a441-kube-api-access-52f69\") pod \"authorino-operator-75cc8d5cb7-xrjrb\" (UID: \"7dc02ebc-a14d-4bb2-81a3-db3512a6a441\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:20.194282 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:20.194215    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:20.331942 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:20.331907    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"]
Jul 30 03:51:20.335153 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:20.335121    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc02ebc_a14d_4bb2_81a3_db3512a6a441.slice/crio-b6b45163dc96ab64ad122f581c55a852cd9ac66efd3e65e7f394135da0cd1c72 WatchSource:0}: Error finding container b6b45163dc96ab64ad122f581c55a852cd9ac66efd3e65e7f394135da0cd1c72: Status 404 returned error can't find the container with id b6b45163dc96ab64ad122f581c55a852cd9ac66efd3e65e7f394135da0cd1c72
Jul 30 03:51:20.429090 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:20.429061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb" event={"ID":"7dc02ebc-a14d-4bb2-81a3-db3512a6a441","Type":"ContainerStarted","Data":"b6b45163dc96ab64ad122f581c55a852cd9ac66efd3e65e7f394135da0cd1c72"}
Jul 30 03:51:23.444467 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:23.444428    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb" event={"ID":"7dc02ebc-a14d-4bb2-81a3-db3512a6a441","Type":"ContainerStarted","Data":"88f53637510a9e27842886f60afa72a6a0e17ee201d15e102cc3ead345f90b13"}
Jul 30 03:51:23.444927 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:23.444679    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:23.476396 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:23.476345    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb" podStartSLOduration=2.3172355590000002 podStartE2EDuration="4.476330905s" totalImagesPullingTime="2.159095346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:23.468300449 +0000 UTC m=+376.285236926" watchObservedRunningTime="2026-07-30 03:51:23.476330905 +0000 UTC m=+376.293267381"
Jul 30 03:51:24.760592 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.760555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"]
Jul 30 03:51:24.763732 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.763715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.767156 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.767130    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhdjg\""
Jul 30 03:51:24.775459 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.775436    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"]
Jul 30 03:51:24.828604 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.828559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.828789 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.828631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6nq\" (UniqueName: \"kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.929323 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.929287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.929485 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.929339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6nq\" (UniqueName: \"kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.929702 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.929682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:24.941417 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:24.941383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6nq\" (UniqueName: \"kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:25.074230 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:25.074127    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:25.202834 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:25.202810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"]
Jul 30 03:51:25.205155 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:25.205117    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5a8a99_8550_4283_b54c_effd1643a3dc.slice/crio-6e6483dce0dd514820a07a7864661ca0a4a15d0992ae5ee195cc050b13f0cebd WatchSource:0}: Error finding container 6e6483dce0dd514820a07a7864661ca0a4a15d0992ae5ee195cc050b13f0cebd: Status 404 returned error can't find the container with id 6e6483dce0dd514820a07a7864661ca0a4a15d0992ae5ee195cc050b13f0cebd
Jul 30 03:51:25.454562 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:25.454446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb" event={"ID":"8a5a8a99-8550-4283-b54c-effd1643a3dc","Type":"ContainerStarted","Data":"6e6483dce0dd514820a07a7864661ca0a4a15d0992ae5ee195cc050b13f0cebd"}
Jul 30 03:51:26.410540 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:26.410495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rctsq"
Jul 30 03:51:29.426780 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:29.426740    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:30.480397 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:30.480361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb" event={"ID":"8a5a8a99-8550-4283-b54c-effd1643a3dc","Type":"ContainerStarted","Data":"fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258"}
Jul 30 03:51:30.480785 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:30.480485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:30.506745 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:30.506685    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb" podStartSLOduration=1.873881027 podStartE2EDuration="6.506662686s" totalImagesPullingTime="4.632781659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:30.50511526 +0000 UTC m=+383.322051740" watchObservedRunningTime="2026-07-30 03:51:30.506662686 +0000 UTC m=+383.323599161"
Jul 30 03:51:32.336580 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.336550    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"]
Jul 30 03:51:32.339835 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.339819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.342286 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.342256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 30 03:51:32.342422 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.342385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 30 03:51:32.342547 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.342506    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7jnsd\""
Jul 30 03:51:32.347106 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.346846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"]
Jul 30 03:51:32.387689 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.387665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de6d0192-055a-438f-9845-dea2db2937a9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.387790 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.387700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmplk\" (UniqueName: \"kubernetes.io/projected/de6d0192-055a-438f-9845-dea2db2937a9-kube-api-access-tmplk\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.387790 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.387738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.488213 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.488190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.488339 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.488237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de6d0192-055a-438f-9845-dea2db2937a9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.488339 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.488261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmplk\" (UniqueName: \"kubernetes.io/projected/de6d0192-055a-438f-9845-dea2db2937a9-kube-api-access-tmplk\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.488339 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:32.488315    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 30 03:51:32.488489 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:32.488394    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert podName:de6d0192-055a-438f-9845-dea2db2937a9 nodeName:}" failed. No retries permitted until 2026-07-30 03:51:32.988377418 +0000 UTC m=+385.805313872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-c22nh" (UID: "de6d0192-055a-438f-9845-dea2db2937a9") : secret "plugin-serving-cert" not found
Jul 30 03:51:32.488863 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.488844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de6d0192-055a-438f-9845-dea2db2937a9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.496533 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.496508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmplk\" (UniqueName: \"kubernetes.io/projected/de6d0192-055a-438f-9845-dea2db2937a9-kube-api-access-tmplk\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.991382 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.991349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:32.993935 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:32.993896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de6d0192-055a-438f-9845-dea2db2937a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c22nh\" (UID: \"de6d0192-055a-438f-9845-dea2db2937a9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:33.249841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:33.249772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"
Jul 30 03:51:33.373394 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:33.373361    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh"]
Jul 30 03:51:33.375156 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:33.375126    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde6d0192_055a_438f_9845_dea2db2937a9.slice/crio-a0feac35185f9e0e72c31d44a6935783163dcb8027e77c803b9945525bd43bf6 WatchSource:0}: Error finding container a0feac35185f9e0e72c31d44a6935783163dcb8027e77c803b9945525bd43bf6: Status 404 returned error can't find the container with id a0feac35185f9e0e72c31d44a6935783163dcb8027e77c803b9945525bd43bf6
Jul 30 03:51:33.493340 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:33.493312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh" event={"ID":"de6d0192-055a-438f-9845-dea2db2937a9","Type":"ContainerStarted","Data":"a0feac35185f9e0e72c31d44a6935783163dcb8027e77c803b9945525bd43bf6"}
Jul 30 03:51:34.451395 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:34.451362    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xrjrb"
Jul 30 03:51:41.486797 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:41.486762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:43.153474 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.153427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"]
Jul 30 03:51:43.153939 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.153709    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" containerName="manager" containerID="cri-o://fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258" gracePeriod=2
Jul 30 03:51:43.164588 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.164486    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"]
Jul 30 03:51:43.167118 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.167075    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8a5a8a99-8550-4283-b54c-effd1643a3dc\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:43.174057 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.174027    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:51:43.174566 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.174545    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" containerName="manager"
Jul 30 03:51:43.174566 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.174565    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" containerName="manager"
Jul 30 03:51:43.174719 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.174647    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" containerName="manager"
Jul 30 03:51:43.184990 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.184966    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.187991 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.187959    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:43.191784 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.191604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"]
Jul 30 03:51:43.192450 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.192046    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" containerName="manager" containerID="cri-o://2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f" gracePeriod=2
Jul 30 03:51:43.193809 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.193700    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:51:43.202594 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.202566    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"]
Jul 30 03:51:43.217791 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.217739    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"]
Jul 30 03:51:43.218250 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.218230    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" containerName="manager"
Jul 30 03:51:43.218250 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.218248    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" containerName="manager"
Jul 30 03:51:43.218396 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.218322    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" containerName="manager"
Jul 30 03:51:43.219580 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.219515    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:43.221573 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.221549    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:43.221692 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.221606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:43.223740 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.223715    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:43.233714 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.233648    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"]
Jul 30 03:51:43.252041 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:43.251998    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:43.286588 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.286555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwsl\" (UniqueName: \"kubernetes.io/projected/321a2104-8ee0-4d45-86af-a89aa3e9be2d-kube-api-access-5bwsl\") pod \"limitador-operator-controller-manager-859d7b78bd-psgzj\" (UID: \"321a2104-8ee0-4d45-86af-a89aa3e9be2d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:43.286725 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.286638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s782\" (UniqueName: \"kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.286725 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.286687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.387216 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.387177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.387393 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.387266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwsl\" (UniqueName: \"kubernetes.io/projected/321a2104-8ee0-4d45-86af-a89aa3e9be2d-kube-api-access-5bwsl\") pod \"limitador-operator-controller-manager-859d7b78bd-psgzj\" (UID: \"321a2104-8ee0-4d45-86af-a89aa3e9be2d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:43.387393 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.387313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s782\" (UniqueName: \"kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.387677 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.387643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.401000 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.400974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwsl\" (UniqueName: \"kubernetes.io/projected/321a2104-8ee0-4d45-86af-a89aa3e9be2d-kube-api-access-5bwsl\") pod \"limitador-operator-controller-manager-859d7b78bd-psgzj\" (UID: \"321a2104-8ee0-4d45-86af-a89aa3e9be2d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:43.402277 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.402242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s782\" (UniqueName: \"kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g5tml\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.606683 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.606648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:43.613786 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:43.613750    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:55.071201 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.070767    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.072102 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.072074    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"]
Jul 30 03:51:55.074279 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.073161    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.077841 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.077821    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:55.078492 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:55.078465    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321a2104_8ee0_4d45_86af_a89aa3e9be2d.slice/crio-6d08d10f308ea8a8b07d9b3575b87ab6b5cf0559f9dcec036f41329d842a1786 WatchSource:0}: Error finding container 6d08d10f308ea8a8b07d9b3575b87ab6b5cf0559f9dcec036f41329d842a1786: Status 404 returned error can't find the container with id 6d08d10f308ea8a8b07d9b3575b87ab6b5cf0559f9dcec036f41329d842a1786
Jul 30 03:51:55.080054 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.080029    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.081816 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.081777    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:55.095283 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.095264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:51:55.098401 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:51:55.098378    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0833b3_9f55_4b78_9ec8_864640cb5a09.slice/crio-fb5322806aa41629cdb78bb9b0fa45c5578197c118dd62d9dd67024b0779d905 WatchSource:0}: Error finding container fb5322806aa41629cdb78bb9b0fa45c5578197c118dd62d9dd67024b0779d905: Status 404 returned error can't find the container with id fb5322806aa41629cdb78bb9b0fa45c5578197c118dd62d9dd67024b0779d905
Jul 30 03:51:55.185423 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.185396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume\") pod \"8a5a8a99-8550-4283-b54c-effd1643a3dc\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") "
Jul 30 03:51:55.185620 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.185485    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw6nq\" (UniqueName: \"kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq\") pod \"8a5a8a99-8550-4283-b54c-effd1643a3dc\" (UID: \"8a5a8a99-8550-4283-b54c-effd1643a3dc\") "
Jul 30 03:51:55.185620 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.185508    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p57xk\" (UniqueName: \"kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk\") pod \"e6c1a66c-e8fc-45ff-8bca-ac998916ace3\" (UID: \"e6c1a66c-e8fc-45ff-8bca-ac998916ace3\") "
Jul 30 03:51:55.185991 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.185961    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8a5a8a99-8550-4283-b54c-effd1643a3dc" (UID: "8a5a8a99-8550-4283-b54c-effd1643a3dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:51:55.187612 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.187585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk" (OuterVolumeSpecName: "kube-api-access-p57xk") pod "e6c1a66c-e8fc-45ff-8bca-ac998916ace3" (UID: "e6c1a66c-e8fc-45ff-8bca-ac998916ace3"). InnerVolumeSpecName "kube-api-access-p57xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:51:55.187706 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.187625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq" (OuterVolumeSpecName: "kube-api-access-rw6nq") pod "8a5a8a99-8550-4283-b54c-effd1643a3dc" (UID: "8a5a8a99-8550-4283-b54c-effd1643a3dc"). InnerVolumeSpecName "kube-api-access-rw6nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:51:55.286368 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.286333    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a5a8a99-8550-4283-b54c-effd1643a3dc-extensions-socket-volume\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:51:55.286368 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.286363    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw6nq\" (UniqueName: \"kubernetes.io/projected/8a5a8a99-8550-4283-b54c-effd1643a3dc-kube-api-access-rw6nq\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:51:55.286368 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.286372    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57xk\" (UniqueName: \"kubernetes.io/projected/e6c1a66c-e8fc-45ff-8bca-ac998916ace3-kube-api-access-p57xk\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:51:55.610545 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.610488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh" event={"ID":"de6d0192-055a-438f-9845-dea2db2937a9","Type":"ContainerStarted","Data":"081e48bf2e8be08dd499c1a8ffd22c5e7a7e75ad95d006fbe653831840cd6d3e"}
Jul 30 03:51:55.612079 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.612051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj" event={"ID":"321a2104-8ee0-4d45-86af-a89aa3e9be2d","Type":"ContainerStarted","Data":"932434c07d50df94a0e1c3bf9f02fdb5889a0297d2902714343096431ccc3810"}
Jul 30 03:51:55.612208 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.612095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj" event={"ID":"321a2104-8ee0-4d45-86af-a89aa3e9be2d","Type":"ContainerStarted","Data":"6d08d10f308ea8a8b07d9b3575b87ab6b5cf0559f9dcec036f41329d842a1786"}
Jul 30 03:51:55.612208 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.612127    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:51:55.612664 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.612631    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:55.613366 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.613346    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a5a8a99-8550-4283-b54c-effd1643a3dc" containerID="fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258" exitCode=0
Jul 30 03:51:55.613433 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.613378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:55.613477 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.613437    2562 scope.go:117] "RemoveContainer" containerID="fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258"
Jul 30 03:51:55.614434 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.614354    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.614675 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.614653    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" containerID="2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f" exitCode=0
Jul 30 03:51:55.614740 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.614725    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.616343 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.616319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" event={"ID":"fd0833b3-9f55-4b78-9ec8-864640cb5a09","Type":"ContainerStarted","Data":"d1e3317f93e432197b80b04dd5857329b18a1357e5a5384e470a5bb45c324a0b"}
Jul 30 03:51:55.616448 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.616365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" event={"ID":"fd0833b3-9f55-4b78-9ec8-864640cb5a09","Type":"ContainerStarted","Data":"fb5322806aa41629cdb78bb9b0fa45c5578197c118dd62d9dd67024b0779d905"}
Jul 30 03:51:55.616448 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.616404    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:55.622938 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.622923    2562 scope.go:117] "RemoveContainer" containerID="fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258"
Jul 30 03:51:55.623175 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.623159    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258\": container with ID starting with fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258 not found: ID does not exist" containerID="fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258"
Jul 30 03:51:55.623251 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.623187    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258"} err="failed to get container status \"fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258\": rpc error: code = NotFound desc = could not find container \"fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258\": container with ID starting with fb3bf185552c8cc932be542c133384fe3de43332ac1f68d860b338a1f9363258 not found: ID does not exist"
Jul 30 03:51:55.623251 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.623209    2562 scope.go:117] "RemoveContainer" containerID="2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f"
Jul 30 03:51:55.629882 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.629845    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c22nh" podStartSLOduration=1.99662285 podStartE2EDuration="23.629834084s" totalImagesPullingTime="21.633211234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:55.629586941 +0000 UTC m=+408.446523418" watchObservedRunningTime="2026-07-30 03:51:55.629834084 +0000 UTC m=+408.446770552"
Jul 30 03:51:55.631142 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.631124    2562 scope.go:117] "RemoveContainer" containerID="2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f"
Jul 30 03:51:55.631406 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.631381    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f\": container with ID starting with 2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f not found: ID does not exist" containerID="2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f"
Jul 30 03:51:55.631464 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.631415    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f"} err="failed to get container status \"2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f\": rpc error: code = NotFound desc = could not find container \"2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f\": container with ID starting with 2dea68233af81bd3800abbfa17ca05d63976258700ab25718ae57ed09b65536f not found: ID does not exist"
Jul 30 03:51:55.631504 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.631467    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qqgm4\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qqgm4"
Jul 30 03:51:55.653321 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.653274    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" podStartSLOduration=12.653258651 podStartE2EDuration="12.653258651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:55.6516214 +0000 UTC m=+408.468558058" watchObservedRunningTime="2026-07-30 03:51:55.653258651 +0000 UTC m=+408.470195178"
Jul 30 03:51:55.653470 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:51:55.653439    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mwvkb\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mwvkb"
Jul 30 03:51:55.672532 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.672469    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj" podStartSLOduration=12.672454986 podStartE2EDuration="12.672454986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:51:55.670666355 +0000 UTC m=+408.487602831" watchObservedRunningTime="2026-07-30 03:51:55.672454986 +0000 UTC m=+408.489391462"
Jul 30 03:51:55.784683 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.784650    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5a8a99-8550-4283-b54c-effd1643a3dc" path="/var/lib/kubelet/pods/8a5a8a99-8550-4283-b54c-effd1643a3dc/volumes"
Jul 30 03:51:55.784991 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:55.784978    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c1a66c-e8fc-45ff-8bca-ac998916ace3" path="/var/lib/kubelet/pods/e6c1a66c-e8fc-45ff-8bca-ac998916ace3/volumes"
Jul 30 03:51:59.397957 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.397924    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:51:59.398346 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.398092    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" podUID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" containerName="manager" containerID="cri-o://d1e3317f93e432197b80b04dd5857329b18a1357e5a5384e470a5bb45c324a0b" gracePeriod=10
Jul 30 03:51:59.637966 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.637931    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" containerID="d1e3317f93e432197b80b04dd5857329b18a1357e5a5384e470a5bb45c324a0b" exitCode=0
Jul 30 03:51:59.638086 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.637980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" event={"ID":"fd0833b3-9f55-4b78-9ec8-864640cb5a09","Type":"ContainerDied","Data":"d1e3317f93e432197b80b04dd5857329b18a1357e5a5384e470a5bb45c324a0b"}
Jul 30 03:51:59.653752 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.653677    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:51:59.667442 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.667414    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 03:51:59.667768 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.667755    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" containerName="manager"
Jul 30 03:51:59.667835 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.667770    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" containerName="manager"
Jul 30 03:51:59.667872 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.667837    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" containerName="manager"
Jul 30 03:51:59.670464 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.670446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.683768 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.683745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 03:51:59.722282 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.722256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume\") pod \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") "
Jul 30 03:51:59.722408 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.722295    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s782\" (UniqueName: \"kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782\") pod \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\" (UID: \"fd0833b3-9f55-4b78-9ec8-864640cb5a09\") "
Jul 30 03:51:59.722672 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.722643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fd0833b3-9f55-4b78-9ec8-864640cb5a09" (UID: "fd0833b3-9f55-4b78-9ec8-864640cb5a09"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 03:51:59.724285 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.724263    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782" (OuterVolumeSpecName: "kube-api-access-9s782") pod "fd0833b3-9f55-4b78-9ec8-864640cb5a09" (UID: "fd0833b3-9f55-4b78-9ec8-864640cb5a09"). InnerVolumeSpecName "kube-api-access-9s782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:51:59.823200 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.823164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548sj\" (UniqueName: \"kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.823339 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.823207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.823339 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.823261    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd0833b3-9f55-4b78-9ec8-864640cb5a09-extensions-socket-volume\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:51:59.823339 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.823271    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s782\" (UniqueName: \"kubernetes.io/projected/fd0833b3-9f55-4b78-9ec8-864640cb5a09-kube-api-access-9s782\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:51:59.924412 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.924331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548sj\" (UniqueName: \"kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.924412 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.924391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.924857 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.924836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.936166 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.936138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-548sj\" (UniqueName: \"kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj\") pod \"kuadrant-operator-controller-manager-86878c769f-zr449\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:51:59.980793 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:51:59.980759    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:52:00.112173 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.112148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 03:52:00.114065 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:52:00.114036    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbcefeb_04a6_4aa0_b94f_13481e469d11.slice/crio-b45e654497a24c2fb192b554a770db6052c798db1745943c8952f0cca33ca152 WatchSource:0}: Error finding container b45e654497a24c2fb192b554a770db6052c798db1745943c8952f0cca33ca152: Status 404 returned error can't find the container with id b45e654497a24c2fb192b554a770db6052c798db1745943c8952f0cca33ca152
Jul 30 03:52:00.647892 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.647855    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" event={"ID":"8fbcefeb-04a6-4aa0-b94f-13481e469d11","Type":"ContainerStarted","Data":"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"}
Jul 30 03:52:00.648424 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.647921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" event={"ID":"8fbcefeb-04a6-4aa0-b94f-13481e469d11","Type":"ContainerStarted","Data":"b45e654497a24c2fb192b554a770db6052c798db1745943c8952f0cca33ca152"}
Jul 30 03:52:00.648424 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.647969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:52:00.649430 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.649408    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"
Jul 30 03:52:00.649562 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.649416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml" event={"ID":"fd0833b3-9f55-4b78-9ec8-864640cb5a09","Type":"ContainerDied","Data":"fb5322806aa41629cdb78bb9b0fa45c5578197c118dd62d9dd67024b0779d905"}
Jul 30 03:52:00.649562 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.649473    2562 scope.go:117] "RemoveContainer" containerID="d1e3317f93e432197b80b04dd5857329b18a1357e5a5384e470a5bb45c324a0b"
Jul 30 03:52:00.669994 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.669946    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" podStartSLOduration=1.669931144 podStartE2EDuration="1.669931144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:52:00.668379908 +0000 UTC m=+413.485316386" watchObservedRunningTime="2026-07-30 03:52:00.669931144 +0000 UTC m=+413.486867621"
Jul 30 03:52:00.682285 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.682256    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:52:00.688765 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:00.688737    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g5tml"]
Jul 30 03:52:01.785211 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:01.785179    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0833b3-9f55-4b78-9ec8-864640cb5a09" path="/var/lib/kubelet/pods/fd0833b3-9f55-4b78-9ec8-864640cb5a09/volumes"
Jul 30 03:52:06.625961 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:06.625922    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgzj"
Jul 30 03:52:11.658412 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:11.658372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 03:52:31.425279 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.425199    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:52:31.428811 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.428788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.431194 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.431169    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 30 03:52:31.435285 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.435262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:52:31.459038 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.459015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:52:31.460688 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.460666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/530be5d0-8d48-499a-976d-0df89dec736e-config-file\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.460777 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.460695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpt96\" (UniqueName: \"kubernetes.io/projected/530be5d0-8d48-499a-976d-0df89dec736e-kube-api-access-qpt96\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.561476 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.561427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/530be5d0-8d48-499a-976d-0df89dec736e-config-file\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.561651 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.561482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpt96\" (UniqueName: \"kubernetes.io/projected/530be5d0-8d48-499a-976d-0df89dec736e-kube-api-access-qpt96\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.562124 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.562100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/530be5d0-8d48-499a-976d-0df89dec736e-config-file\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.569802 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.569773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpt96\" (UniqueName: \"kubernetes.io/projected/530be5d0-8d48-499a-976d-0df89dec736e-kube-api-access-qpt96\") pod \"limitador-limitador-54d6d988d5-q5bzq\" (UID: \"530be5d0-8d48-499a-976d-0df89dec736e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.740380 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.740336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:31.872828 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:31.872804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:52:31.874941 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:52:31.874912    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530be5d0_8d48_499a_976d_0df89dec736e.slice/crio-3bbccf63c08465d61c1d8805f4d54a3711a7995b10e4be26d906a65195f61377 WatchSource:0}: Error finding container 3bbccf63c08465d61c1d8805f4d54a3711a7995b10e4be26d906a65195f61377: Status 404 returned error can't find the container with id 3bbccf63c08465d61c1d8805f4d54a3711a7995b10e4be26d906a65195f61377
Jul 30 03:52:32.018354 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.018278    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:52:32.022391 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.022375    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:52:32.024831 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.024801    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lzq79\""
Jul 30 03:52:32.027512 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.027488    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:52:32.067166 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.067140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8s9\" (UniqueName: \"kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9\") pod \"authorino-75d556bd77-d72ms\" (UID: \"294bfbe1-d23e-420e-b71d-1934ac18a4aa\") " pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:52:32.168291 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.168262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8s9\" (UniqueName: \"kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9\") pod \"authorino-75d556bd77-d72ms\" (UID: \"294bfbe1-d23e-420e-b71d-1934ac18a4aa\") " pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:52:32.175984 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.175957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8s9\" (UniqueName: \"kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9\") pod \"authorino-75d556bd77-d72ms\" (UID: \"294bfbe1-d23e-420e-b71d-1934ac18a4aa\") " pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:52:32.343956 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.343880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:52:32.510704 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.510642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:52:32.513831 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:52:32.513796    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod294bfbe1_d23e_420e_b71d_1934ac18a4aa.slice/crio-d89e57412bf76358262654d19bd330ab38a941cc30022f2a11e603d60deafdce WatchSource:0}: Error finding container d89e57412bf76358262654d19bd330ab38a941cc30022f2a11e603d60deafdce: Status 404 returned error can't find the container with id d89e57412bf76358262654d19bd330ab38a941cc30022f2a11e603d60deafdce
Jul 30 03:52:32.802283 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.802249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d72ms" event={"ID":"294bfbe1-d23e-420e-b71d-1934ac18a4aa","Type":"ContainerStarted","Data":"d89e57412bf76358262654d19bd330ab38a941cc30022f2a11e603d60deafdce"}
Jul 30 03:52:32.803499 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:32.803467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq" event={"ID":"530be5d0-8d48-499a-976d-0df89dec736e","Type":"ContainerStarted","Data":"3bbccf63c08465d61c1d8805f4d54a3711a7995b10e4be26d906a65195f61377"}
Jul 30 03:52:36.830486 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:36.830445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d72ms" event={"ID":"294bfbe1-d23e-420e-b71d-1934ac18a4aa","Type":"ContainerStarted","Data":"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"}
Jul 30 03:52:36.832126 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:36.832096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq" event={"ID":"530be5d0-8d48-499a-976d-0df89dec736e","Type":"ContainerStarted","Data":"67e7a88788d8df1095424c59c68bea07f1a8887ff4cbbd61e92d8d86bb45962b"}
Jul 30 03:52:36.832250 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:36.832162    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:52:36.847045 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:36.847000    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-d72ms" podStartSLOduration=1.020610635 podStartE2EDuration="4.846987193s" totalImagesPullingTime="3.826376558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:52:36.84529706 +0000 UTC m=+449.662233537" watchObservedRunningTime="2026-07-30 03:52:36.846987193 +0000 UTC m=+449.663923669"
Jul 30 03:52:36.862766 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:36.862715    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq" podStartSLOduration=1.442861354 podStartE2EDuration="5.862697119s" totalImagesPullingTime="4.419835765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:52:36.861194395 +0000 UTC m=+449.678130872" watchObservedRunningTime="2026-07-30 03:52:36.862697119 +0000 UTC m=+449.679633598"
Jul 30 03:52:47.837215 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:52:47.837183    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"
Jul 30 03:53:02.251616 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.251581    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:02.254670 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.254650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:02.264953 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.264922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:02.300247 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.300220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8zp\" (UniqueName: \"kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp\") pod \"authorino-5d4b67c446-2x8rp\" (UID: \"0c63ef2a-b200-48ed-969c-9f50cda26cfe\") " pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:02.401133 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.401104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8zp\" (UniqueName: \"kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp\") pod \"authorino-5d4b67c446-2x8rp\" (UID: \"0c63ef2a-b200-48ed-969c-9f50cda26cfe\") " pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:02.409980 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.409954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8zp\" (UniqueName: \"kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp\") pod \"authorino-5d4b67c446-2x8rp\" (UID: \"0c63ef2a-b200-48ed-969c-9f50cda26cfe\") " pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:02.491363 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.491340    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:02.491550 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.491538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:02.517476 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.517450    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-799688bc8b-bq9lk"]
Jul 30 03:53:02.521150 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.521132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.527108 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.527085    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-799688bc8b-bq9lk"]
Jul 30 03:53:02.602108 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.602084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvz8s\" (UniqueName: \"kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s\") pod \"authorino-799688bc8b-bq9lk\" (UID: \"ef7bbcb0-9211-4ca1-a60b-e08a309412f4\") " pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.623130 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.623107    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:02.625637 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:53:02.625603    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c63ef2a_b200_48ed_969c_9f50cda26cfe.slice/crio-d56ff5f380a6129b131f35df872b4044040d37730f11a0f2d36353ad7f93b568 WatchSource:0}: Error finding container d56ff5f380a6129b131f35df872b4044040d37730f11a0f2d36353ad7f93b568: Status 404 returned error can't find the container with id d56ff5f380a6129b131f35df872b4044040d37730f11a0f2d36353ad7f93b568
Jul 30 03:53:02.654924 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.654899    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-799688bc8b-bq9lk"]
Jul 30 03:53:02.655154 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:53:02.655136    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fvz8s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-799688bc8b-bq9lk" podUID="ef7bbcb0-9211-4ca1-a60b-e08a309412f4"
Jul 30 03:53:02.682850 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.682824    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:53:02.686204 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.686187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.688803 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.688784    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 30 03:53:02.693606 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.693584    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:53:02.702755 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.702735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvz8s\" (UniqueName: \"kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s\") pod \"authorino-799688bc8b-bq9lk\" (UID: \"ef7bbcb0-9211-4ca1-a60b-e08a309412f4\") " pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.710136 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.710118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvz8s\" (UniqueName: \"kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s\") pod \"authorino-799688bc8b-bq9lk\" (UID: \"ef7bbcb0-9211-4ca1-a60b-e08a309412f4\") " pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.804009 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.803951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mcc\" (UniqueName: \"kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.804009 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.803993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.904289 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.904263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.904386 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.904318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mcc\" (UniqueName: \"kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.906615 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.906597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.912255 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.912233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mcc\" (UniqueName: \"kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc\") pod \"authorino-fd95ff674-ld22v\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") " pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:02.946869 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.946849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.946950 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.946839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" event={"ID":"0c63ef2a-b200-48ed-969c-9f50cda26cfe","Type":"ContainerStarted","Data":"d56ff5f380a6129b131f35df872b4044040d37730f11a0f2d36353ad7f93b568"}
Jul 30 03:53:02.953650 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.953633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:02.996687 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:02.996668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:53:03.005365 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.005345    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvz8s\" (UniqueName: \"kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s\") pod \"ef7bbcb0-9211-4ca1-a60b-e08a309412f4\" (UID: \"ef7bbcb0-9211-4ca1-a60b-e08a309412f4\") "
Jul 30 03:53:03.007608 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.007587    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s" (OuterVolumeSpecName: "kube-api-access-fvz8s") pod "ef7bbcb0-9211-4ca1-a60b-e08a309412f4" (UID: "ef7bbcb0-9211-4ca1-a60b-e08a309412f4"). InnerVolumeSpecName "kube-api-access-fvz8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:53:03.106254 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.106224    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvz8s\" (UniqueName: \"kubernetes.io/projected/ef7bbcb0-9211-4ca1-a60b-e08a309412f4-kube-api-access-fvz8s\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:53:03.125047 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.125025    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:53:03.126805 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:53:03.126781    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda926dab0_8392_455c_9c95_4d31023a2853.slice/crio-27ba811b90b07d031730115ef57cdfc70413464ba53c02415a9c815cf2415960 WatchSource:0}: Error finding container 27ba811b90b07d031730115ef57cdfc70413464ba53c02415a9c815cf2415960: Status 404 returned error can't find the container with id 27ba811b90b07d031730115ef57cdfc70413464ba53c02415a9c815cf2415960
Jul 30 03:53:03.953471 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.953436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd95ff674-ld22v" event={"ID":"a926dab0-8392-455c-9c95-4d31023a2853","Type":"ContainerStarted","Data":"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"}
Jul 30 03:53:03.953884 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.953480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd95ff674-ld22v" event={"ID":"a926dab0-8392-455c-9c95-4d31023a2853","Type":"ContainerStarted","Data":"27ba811b90b07d031730115ef57cdfc70413464ba53c02415a9c815cf2415960"}
Jul 30 03:53:03.955091 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.955065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" podUID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" containerName="authorino" containerID="cri-o://724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e" gracePeriod=30
Jul 30 03:53:03.955195 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.955080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" event={"ID":"0c63ef2a-b200-48ed-969c-9f50cda26cfe","Type":"ContainerStarted","Data":"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"}
Jul 30 03:53:03.955195 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.955078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799688bc8b-bq9lk"
Jul 30 03:53:03.970177 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.970138    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fd95ff674-ld22v" podStartSLOduration=1.635145134 podStartE2EDuration="1.970124784s" totalImagesPullingTime="334.97965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:53:03.968792394 +0000 UTC m=+476.785728872" watchObservedRunningTime="2026-07-30 03:53:03.970124784 +0000 UTC m=+476.787061258"
Jul 30 03:53:03.983855 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.983805    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" podStartSLOduration=1.451046188 podStartE2EDuration="1.983787548s" totalImagesPullingTime="532.74136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:53:03.983026082 +0000 UTC m=+476.799962554" watchObservedRunningTime="2026-07-30 03:53:03.983787548 +0000 UTC m=+476.800724024"
Jul 30 03:53:03.997423 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.997399    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:53:03.998036 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:03.997663    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-d72ms" podUID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" containerName="authorino" containerID="cri-o://b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c" gracePeriod=30
Jul 30 03:53:04.012318 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.012294    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-799688bc8b-bq9lk"]
Jul 30 03:53:04.018584 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.018551    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-799688bc8b-bq9lk"]
Jul 30 03:53:04.239591 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.239566    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:04.271256 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.271206    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:53:04.316858 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.316833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p8zp\" (UniqueName: \"kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp\") pod \"0c63ef2a-b200-48ed-969c-9f50cda26cfe\" (UID: \"0c63ef2a-b200-48ed-969c-9f50cda26cfe\") "
Jul 30 03:53:04.318799 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.318776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp" (OuterVolumeSpecName: "kube-api-access-2p8zp") pod "0c63ef2a-b200-48ed-969c-9f50cda26cfe" (UID: "0c63ef2a-b200-48ed-969c-9f50cda26cfe"). InnerVolumeSpecName "kube-api-access-2p8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:53:04.418132 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.418110    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck8s9\" (UniqueName: \"kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9\") pod \"294bfbe1-d23e-420e-b71d-1934ac18a4aa\" (UID: \"294bfbe1-d23e-420e-b71d-1934ac18a4aa\") "
Jul 30 03:53:04.418297 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.418284    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8zp\" (UniqueName: \"kubernetes.io/projected/0c63ef2a-b200-48ed-969c-9f50cda26cfe-kube-api-access-2p8zp\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:53:04.420110 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.420086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9" (OuterVolumeSpecName: "kube-api-access-ck8s9") pod "294bfbe1-d23e-420e-b71d-1934ac18a4aa" (UID: "294bfbe1-d23e-420e-b71d-1934ac18a4aa"). InnerVolumeSpecName "kube-api-access-ck8s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:53:04.519429 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.519408    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8s9\" (UniqueName: \"kubernetes.io/projected/294bfbe1-d23e-420e-b71d-1934ac18a4aa-kube-api-access-ck8s9\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:53:04.960815 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.960778    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" containerID="724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e" exitCode=0
Jul 30 03:53:04.961259 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.960819    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2x8rp"
Jul 30 03:53:04.961259 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.960864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" event={"ID":"0c63ef2a-b200-48ed-969c-9f50cda26cfe","Type":"ContainerDied","Data":"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"}
Jul 30 03:53:04.961259 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.960902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2x8rp" event={"ID":"0c63ef2a-b200-48ed-969c-9f50cda26cfe","Type":"ContainerDied","Data":"d56ff5f380a6129b131f35df872b4044040d37730f11a0f2d36353ad7f93b568"}
Jul 30 03:53:04.961259 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.960920    2562 scope.go:117] "RemoveContainer" containerID="724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"
Jul 30 03:53:04.962364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.962197    2562 generic.go:356] "Generic (PLEG): container finished" podID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" containerID="b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c" exitCode=0
Jul 30 03:53:04.962364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.962247    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d72ms"
Jul 30 03:53:04.962364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.962286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d72ms" event={"ID":"294bfbe1-d23e-420e-b71d-1934ac18a4aa","Type":"ContainerDied","Data":"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"}
Jul 30 03:53:04.962364 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.962329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d72ms" event={"ID":"294bfbe1-d23e-420e-b71d-1934ac18a4aa","Type":"ContainerDied","Data":"d89e57412bf76358262654d19bd330ab38a941cc30022f2a11e603d60deafdce"}
Jul 30 03:53:04.970011 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.969904    2562 scope.go:117] "RemoveContainer" containerID="724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"
Jul 30 03:53:04.970220 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:53:04.970188    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e\": container with ID starting with 724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e not found: ID does not exist" containerID="724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"
Jul 30 03:53:04.970350 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.970233    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e"} err="failed to get container status \"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e\": rpc error: code = NotFound desc = could not find container \"724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e\": container with ID starting with 724b27e232961afc832ec4a597c5941810754a8a79c8645cbcff4acdb6e4fe0e not found: ID does not exist"
Jul 30 03:53:04.970350 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.970257    2562 scope.go:117] "RemoveContainer" containerID="b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"
Jul 30 03:53:04.978064 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.978047    2562 scope.go:117] "RemoveContainer" containerID="b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"
Jul 30 03:53:04.978337 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:53:04.978318    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c\": container with ID starting with b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c not found: ID does not exist" containerID="b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"
Jul 30 03:53:04.978387 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.978344    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c"} err="failed to get container status \"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c\": rpc error: code = NotFound desc = could not find container \"b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c\": container with ID starting with b4f07faef8b9e11e0fdcabfdf2004aad8e0c4b51f197f332aff926df61c7202c not found: ID does not exist"
Jul 30 03:53:04.983623 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.983602    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:04.986708 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.986687    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2x8rp"]
Jul 30 03:53:04.996070 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.996051    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:53:04.999867 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:04.999848    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d72ms"]
Jul 30 03:53:05.784379 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:05.784348    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" path="/var/lib/kubelet/pods/0c63ef2a-b200-48ed-969c-9f50cda26cfe/volumes"
Jul 30 03:53:05.784696 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:05.784682    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" path="/var/lib/kubelet/pods/294bfbe1-d23e-420e-b71d-1934ac18a4aa/volumes"
Jul 30 03:53:05.784961 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:05.784950    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7bbcb0-9211-4ca1-a60b-e08a309412f4" path="/var/lib/kubelet/pods/ef7bbcb0-9211-4ca1-a60b-e08a309412f4/volumes"
Jul 30 03:53:45.967391 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:53:45.967356    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:54:06.062367 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:54:06.062324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:54:19.562271 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:54:19.562238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:54:24.352873 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:54:24.352837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:54:34.143872 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:54:34.143831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:54:50.357025 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:54:50.356985    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:55:04.548016 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.547981    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"]
Jul 30 03:55:04.548491 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548472    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" containerName="authorino"
Jul 30 03:55:04.548559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548495    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" containerName="authorino"
Jul 30 03:55:04.548559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548536    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" containerName="authorino"
Jul 30 03:55:04.548559 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548545    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" containerName="authorino"
Jul 30 03:55:04.548677 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548636    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c63ef2a-b200-48ed-969c-9f50cda26cfe" containerName="authorino"
Jul 30 03:55:04.548677 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.548652    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="294bfbe1-d23e-420e-b71d-1934ac18a4aa" containerName="authorino"
Jul 30 03:55:04.551868 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.551849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.555581 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.555560    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 03:55:04.555694 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.555601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bwnzj\""
Jul 30 03:55:04.555694 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.555617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 03:55:04.555694 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.555676    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 30 03:55:04.560223 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.560203    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"]
Jul 30 03:55:04.614400 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba7cb01-24b8-4676-8a65-fa07a5536082-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.614618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.614618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsk47\" (UniqueName: \"kubernetes.io/projected/5ba7cb01-24b8-4676-8a65-fa07a5536082-kube-api-access-tsk47\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.614618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.614618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.614618 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.614596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715614 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715758 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsk47\" (UniqueName: \"kubernetes.io/projected/5ba7cb01-24b8-4676-8a65-fa07a5536082-kube-api-access-tsk47\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715758 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715758 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715758 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.715976 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.715791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba7cb01-24b8-4676-8a65-fa07a5536082-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.716098 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.716072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.716161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.716080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.716161 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.716128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.718089 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.718069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba7cb01-24b8-4676-8a65-fa07a5536082-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.718248 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.718232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba7cb01-24b8-4676-8a65-fa07a5536082-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.723289 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.723268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsk47\" (UniqueName: \"kubernetes.io/projected/5ba7cb01-24b8-4676-8a65-fa07a5536082-kube-api-access-tsk47\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-62x7v\" (UID: \"5ba7cb01-24b8-4676-8a65-fa07a5536082\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.861779 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.861691    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:04.992014 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:04.991986    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"]
Jul 30 03:55:04.994726 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:55:04.994695    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ba7cb01_24b8_4676_8a65_fa07a5536082.slice/crio-606eb2fa1b31b6b3f6fcaa5bec51a7f7b6127ff5d3f025c77a8811600eeb74f2 WatchSource:0}: Error finding container 606eb2fa1b31b6b3f6fcaa5bec51a7f7b6127ff5d3f025c77a8811600eeb74f2: Status 404 returned error can't find the container with id 606eb2fa1b31b6b3f6fcaa5bec51a7f7b6127ff5d3f025c77a8811600eeb74f2
Jul 30 03:55:05.516361 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:05.516313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v" event={"ID":"5ba7cb01-24b8-4676-8a65-fa07a5536082","Type":"ContainerStarted","Data":"606eb2fa1b31b6b3f6fcaa5bec51a7f7b6127ff5d3f025c77a8811600eeb74f2"}
Jul 30 03:55:06.522415 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:06.522377    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v" event={"ID":"5ba7cb01-24b8-4676-8a65-fa07a5536082","Type":"ContainerStarted","Data":"c9ba2243504b929ff9a9e1373c132181d9e1663e18c1b004e59e52b8eccfde41"}
Jul 30 03:55:06.540565 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:06.540501    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v" podStartSLOduration=1.400110346 podStartE2EDuration="2.540486258s" totalImagesPullingTime="1.140375912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:55:06.538348443 +0000 UTC m=+599.355284923" watchObservedRunningTime="2026-07-30 03:55:06.540486258 +0000 UTC m=+599.357422726"
Jul 30 03:55:07.247309 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:07.247275    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:55:14.862095 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:14.862058    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:14.862565 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:14.862107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:14.874276 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:14.874251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:15.575333 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:15.575306    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-62x7v"
Jul 30 03:55:57.024822 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.024789    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687744f755-dt7xz"]
Jul 30 03:55:57.028033 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.028012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.035189 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.035163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687744f755-dt7xz"]
Jul 30 03:55:57.114945 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.114912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d5b1ef-16a4-465b-9ae0-2bd1491059be-tls-cert\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.115074 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.114977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqr2\" (UniqueName: \"kubernetes.io/projected/25d5b1ef-16a4-465b-9ae0-2bd1491059be-kube-api-access-qmqr2\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.215742 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.215704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d5b1ef-16a4-465b-9ae0-2bd1491059be-tls-cert\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.215917 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.215788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqr2\" (UniqueName: \"kubernetes.io/projected/25d5b1ef-16a4-465b-9ae0-2bd1491059be-kube-api-access-qmqr2\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.218076 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.218046    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25d5b1ef-16a4-465b-9ae0-2bd1491059be-tls-cert\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.224009 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.223984    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqr2\" (UniqueName: \"kubernetes.io/projected/25d5b1ef-16a4-465b-9ae0-2bd1491059be-kube-api-access-qmqr2\") pod \"authorino-687744f755-dt7xz\" (UID: \"25d5b1ef-16a4-465b-9ae0-2bd1491059be\") " pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.337685 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.337597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687744f755-dt7xz"
Jul 30 03:55:57.466691 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.466667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687744f755-dt7xz"]
Jul 30 03:55:57.469069 ip-10-0-138-44 kubenswrapper[2562]: W0730 03:55:57.469034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25d5b1ef_16a4_465b_9ae0_2bd1491059be.slice/crio-452702b3dcb690e99585baaf4096511e573707419b81ab55334ca9e9aab7644f WatchSource:0}: Error finding container 452702b3dcb690e99585baaf4096511e573707419b81ab55334ca9e9aab7644f: Status 404 returned error can't find the container with id 452702b3dcb690e99585baaf4096511e573707419b81ab55334ca9e9aab7644f
Jul 30 03:55:57.470873 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.470852    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 03:55:57.762002 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:57.761970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687744f755-dt7xz" event={"ID":"25d5b1ef-16a4-465b-9ae0-2bd1491059be","Type":"ContainerStarted","Data":"452702b3dcb690e99585baaf4096511e573707419b81ab55334ca9e9aab7644f"}
Jul 30 03:55:58.770825 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:58.770774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687744f755-dt7xz" event={"ID":"25d5b1ef-16a4-465b-9ae0-2bd1491059be","Type":"ContainerStarted","Data":"720235aa0efce24c122ea8eacf514b48d543c29e8cbc25f11df3b72b58774d91"}
Jul 30 03:55:58.817746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:58.816812    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-687744f755-dt7xz" podStartSLOduration=1.325446031 podStartE2EDuration="1.816793025s" totalImagesPullingTime="491.346994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 03:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 03:55:58.793369159 +0000 UTC m=+651.610305660" watchObservedRunningTime="2026-07-30 03:55:58.816793025 +0000 UTC m=+651.633729502"
Jul 30 03:55:58.817746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:58.817431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:55:58.817746 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:58.817619    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fd95ff674-ld22v" podUID="a926dab0-8392-455c-9c95-4d31023a2853" containerName="authorino" containerID="cri-o://3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9" gracePeriod=30
Jul 30 03:55:59.087899 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.087873    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:55:59.235464 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.235431    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5mcc\" (UniqueName: \"kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc\") pod \"a926dab0-8392-455c-9c95-4d31023a2853\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") "
Jul 30 03:55:59.235639 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.235512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert\") pod \"a926dab0-8392-455c-9c95-4d31023a2853\" (UID: \"a926dab0-8392-455c-9c95-4d31023a2853\") "
Jul 30 03:55:59.237502 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.237472    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc" (OuterVolumeSpecName: "kube-api-access-h5mcc") pod "a926dab0-8392-455c-9c95-4d31023a2853" (UID: "a926dab0-8392-455c-9c95-4d31023a2853"). InnerVolumeSpecName "kube-api-access-h5mcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 03:55:59.245710 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.245683    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a926dab0-8392-455c-9c95-4d31023a2853" (UID: "a926dab0-8392-455c-9c95-4d31023a2853"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 03:55:59.336199 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.336117    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mcc\" (UniqueName: \"kubernetes.io/projected/a926dab0-8392-455c-9c95-4d31023a2853-kube-api-access-h5mcc\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:55:59.336199 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.336147    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a926dab0-8392-455c-9c95-4d31023a2853-tls-cert\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 03:55:59.777727 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.777692    2562 generic.go:356] "Generic (PLEG): container finished" podID="a926dab0-8392-455c-9c95-4d31023a2853" containerID="3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9" exitCode=0
Jul 30 03:55:59.778191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.777748    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd95ff674-ld22v"
Jul 30 03:55:59.778191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.777785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd95ff674-ld22v" event={"ID":"a926dab0-8392-455c-9c95-4d31023a2853","Type":"ContainerDied","Data":"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"}
Jul 30 03:55:59.778191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.777832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd95ff674-ld22v" event={"ID":"a926dab0-8392-455c-9c95-4d31023a2853","Type":"ContainerDied","Data":"27ba811b90b07d031730115ef57cdfc70413464ba53c02415a9c815cf2415960"}
Jul 30 03:55:59.778191 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.777851    2562 scope.go:117] "RemoveContainer" containerID="3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"
Jul 30 03:55:59.800121 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.800100    2562 scope.go:117] "RemoveContainer" containerID="3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"
Jul 30 03:55:59.800424 ip-10-0-138-44 kubenswrapper[2562]: E0730 03:55:59.800396    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9\": container with ID starting with 3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9 not found: ID does not exist" containerID="3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"
Jul 30 03:55:59.800543 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.800443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9"} err="failed to get container status \"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9\": rpc error: code = NotFound desc = could not find container \"3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9\": container with ID starting with 3baed2d3512e7b98d0cd22014afeec0575ae832221ce25234a158ac215230aa9 not found: ID does not exist"
Jul 30 03:55:59.806020 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.805997    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:55:59.809790 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:55:59.809766    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fd95ff674-ld22v"]
Jul 30 03:56:01.785668 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:56:01.785637    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a926dab0-8392-455c-9c95-4d31023a2853" path="/var/lib/kubelet/pods/a926dab0-8392-455c-9c95-4d31023a2853/volumes"
Jul 30 03:56:29.452883 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:56:29.452844    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:56:38.837793 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:56:38.837761    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:56:49.050551 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:56:49.050499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:58:26.651198 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:58:26.651159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:58:39.252997 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:58:39.252915    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:59:44.048135 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:59:44.048103    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 03:59:53.670641 ip-10-0-138-44 kubenswrapper[2562]: I0730 03:59:53.670603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:00:19.754351 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:00:19.754314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:00:29.438412 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:00:29.438376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:00:39.244155 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:00:39.244118    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:00:48.841751 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:00:48.841714    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:01:00.148557 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:01:00.148508    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:01:09.640817 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:01:09.640780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:01:32.641462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:01:32.641383    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:01:45.646920 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:01:45.646890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:02:05.955704 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:02:05.955646    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:02:17.056283 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:02:17.056249    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:02:33.949194 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:02:33.949154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:02:54.441598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:02:54.441564    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:03:10.350852 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:03:10.350817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:03:19.348776 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:03:19.348738    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:03:30.455138 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:03:30.455101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:03:39.744438 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:03:39.744397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:03:49.843536 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:03:49.843494    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:04:11.142297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:04:11.142262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:04:20.355589 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:04:20.355550    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:04:30.544034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:04:30.543948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:05:19.859448 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:19.859408    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 04:05:19.860042 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:19.859638    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" podUID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" containerName="manager" containerID="cri-o://9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053" gracePeriod=10
Jul 30 04:05:20.114867 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.114810    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 04:05:20.220513 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.220486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume\") pod \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") "
Jul 30 04:05:20.220673 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.220563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-548sj\" (UniqueName: \"kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj\") pod \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\" (UID: \"8fbcefeb-04a6-4aa0-b94f-13481e469d11\") "
Jul 30 04:05:20.220803 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.220779    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8fbcefeb-04a6-4aa0-b94f-13481e469d11" (UID: "8fbcefeb-04a6-4aa0-b94f-13481e469d11"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:05:20.222512 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.222476    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj" (OuterVolumeSpecName: "kube-api-access-548sj") pod "8fbcefeb-04a6-4aa0-b94f-13481e469d11" (UID: "8fbcefeb-04a6-4aa0-b94f-13481e469d11"). InnerVolumeSpecName "kube-api-access-548sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:05:20.321314 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.321284    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fbcefeb-04a6-4aa0-b94f-13481e469d11-extensions-socket-volume\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:05:20.321314 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.321310    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-548sj\" (UniqueName: \"kubernetes.io/projected/8fbcefeb-04a6-4aa0-b94f-13481e469d11-kube-api-access-548sj\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:05:20.405869 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.405798    2562 generic.go:356] "Generic (PLEG): container finished" podID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" containerID="9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053" exitCode=0
Jul 30 04:05:20.405869 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.405849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" event={"ID":"8fbcefeb-04a6-4aa0-b94f-13481e469d11","Type":"ContainerDied","Data":"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"}
Jul 30 04:05:20.405869 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.405862    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"
Jul 30 04:05:20.406031 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.405895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449" event={"ID":"8fbcefeb-04a6-4aa0-b94f-13481e469d11","Type":"ContainerDied","Data":"b45e654497a24c2fb192b554a770db6052c798db1745943c8952f0cca33ca152"}
Jul 30 04:05:20.406031 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.405918    2562 scope.go:117] "RemoveContainer" containerID="9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"
Jul 30 04:05:20.415901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.415883    2562 scope.go:117] "RemoveContainer" containerID="9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"
Jul 30 04:05:20.416157 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:05:20.416139    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053\": container with ID starting with 9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053 not found: ID does not exist" containerID="9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"
Jul 30 04:05:20.416212 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.416164    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053"} err="failed to get container status \"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053\": rpc error: code = NotFound desc = could not find container \"9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053\": container with ID starting with 9b1767164a98b171c9004814cfe27123fc8c7aa37371fd361d3e8f0f02a27053 not found: ID does not exist"
Jul 30 04:05:20.427931 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.427907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 04:05:20.430749 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:20.430728    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zr449"]
Jul 30 04:05:21.785353 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:05:21.785312    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" path="/var/lib/kubelet/pods/8fbcefeb-04a6-4aa0-b94f-13481e469d11/volumes"
Jul 30 04:06:20.603737 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.603694    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:06:20.604171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604059    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" containerName="manager"
Jul 30 04:06:20.604171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604072    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" containerName="manager"
Jul 30 04:06:20.604171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604079    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a926dab0-8392-455c-9c95-4d31023a2853" containerName="authorino"
Jul 30 04:06:20.604171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604085    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a926dab0-8392-455c-9c95-4d31023a2853" containerName="authorino"
Jul 30 04:06:20.604296 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604173    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbcefeb-04a6-4aa0-b94f-13481e469d11" containerName="manager"
Jul 30 04:06:20.604296 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.604184    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a926dab0-8392-455c-9c95-4d31023a2853" containerName="authorino"
Jul 30 04:06:20.608270 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.608251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.610706 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.610687    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhdjg\""
Jul 30 04:06:20.616533 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.616454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:06:20.778448 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.778415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.778643 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.778479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgwn\" (UniqueName: \"kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.878900 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.878813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgwn\" (UniqueName: \"kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.878900 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.878883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.879274 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.879252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.888893 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.888874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgwn\" (UniqueName: \"kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn\") pod \"kuadrant-operator-controller-manager-86878c769f-9s8hv\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:20.919052 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:20.919028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:21.055552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:21.055503    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:06:21.058785 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:06:21.058754    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb105999e_78b2_4a14_b0c4_fab7e7caca4b.slice/crio-1eed2e2b68df2ad2912d8fb98877b4f3075fadc6afea258e095bf17569296c61 WatchSource:0}: Error finding container 1eed2e2b68df2ad2912d8fb98877b4f3075fadc6afea258e095bf17569296c61: Status 404 returned error can't find the container with id 1eed2e2b68df2ad2912d8fb98877b4f3075fadc6afea258e095bf17569296c61
Jul 30 04:06:21.705278 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:21.705239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" event={"ID":"b105999e-78b2-4a14-b0c4-fab7e7caca4b","Type":"ContainerStarted","Data":"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"}
Jul 30 04:06:21.705694 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:21.705308    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" event={"ID":"b105999e-78b2-4a14-b0c4-fab7e7caca4b","Type":"ContainerStarted","Data":"1eed2e2b68df2ad2912d8fb98877b4f3075fadc6afea258e095bf17569296c61"}
Jul 30 04:06:21.705694 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:21.705335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:21.725587 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:21.725539    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" podStartSLOduration=1.7255077760000002 podStartE2EDuration="1.725507776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:06:21.723174993 +0000 UTC m=+1274.540111470" watchObservedRunningTime="2026-07-30 04:06:21.725507776 +0000 UTC m=+1274.542444252"
Jul 30 04:06:32.712289 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:32.712256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:06:43.043920 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:43.043886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:06:51.148063 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:06:51.148025    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:07:16.343597 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:07:16.343559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:07:45.444847 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:07:45.444806    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:07:54.543457 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:07:54.543421    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:03.642990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:03.642960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:12.640079 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:12.640037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:22.860271 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:22.860235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:33.345683 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:33.345650    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:44.845726 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:44.845690    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:08:55.844801 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:08:55.844765    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:09:04.641096 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:04.641063    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:09:30.941595 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:30.941555    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:09:38.646683 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:38.646648    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:09:45.352550 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.352488    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:09:45.355969 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.355945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.358768 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.358749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a3a71fd3-kserve-self-signed-certs\""
Jul 30 04:09:45.373175 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.373153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:09:45.515340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.515340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.515586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.515586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hws\" (UniqueName: \"kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.515586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.515690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.515590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.616781 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.616781 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.616781 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617018 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hws\" (UniqueName: \"kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617018 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617018 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.616876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617172 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.617105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617172 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.617140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.617276 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.617189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.619060 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.619029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.619274 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.619256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.624269 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.624245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hws\" (UniqueName: \"kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws\") pod \"e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") " pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.665093 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.665068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:45.796355 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:45.796251    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:09:45.799139 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:09:45.799112    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f84b74_799a_4d6a_bd27_659caa1175d2.slice/crio-15a13d05fb5ae65167ddfcc17bb51ccbdf35870b9579a7427ab33c24e0cf615d WatchSource:0}: Error finding container 15a13d05fb5ae65167ddfcc17bb51ccbdf35870b9579a7427ab33c24e0cf615d: Status 404 returned error can't find the container with id 15a13d05fb5ae65167ddfcc17bb51ccbdf35870b9579a7427ab33c24e0cf615d
Jul 30 04:09:46.672536 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:46.672494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" event={"ID":"57f84b74-799a-4d6a-bd27-659caa1175d2","Type":"ContainerStarted","Data":"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"}
Jul 30 04:09:46.672903 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:46.672556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" event={"ID":"57f84b74-799a-4d6a-bd27-659caa1175d2","Type":"ContainerStarted","Data":"15a13d05fb5ae65167ddfcc17bb51ccbdf35870b9579a7427ab33c24e0cf615d"}
Jul 30 04:09:46.690386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:46.690332    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" podStartSLOduration=1.69031199 podStartE2EDuration="1.69031199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:46.687807812 +0000 UTC m=+1479.504744288" watchObservedRunningTime="2026-07-30 04:09:46.69031199 +0000 UTC m=+1479.507248469"
Jul 30 04:09:49.345148 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:49.345114    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:09:54.446700 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.446662    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:09:54.450543 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.450506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.452868 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.452841    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a3a71fd3-kserve-self-signed-certs\""
Jul 30 04:09:54.457094 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.457072    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:09:54.596068 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.596253 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.596253 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.596253 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.596253 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.596253 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.596240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg74p\" (UniqueName: \"kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697457 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697457 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697684 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697684 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697684 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697684 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg74p\" (UniqueName: \"kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697908 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697964 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.697964 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.697925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.699760 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.699737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.700128 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.700104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.704799 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.704779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg74p\" (UniqueName: \"kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p\") pod \"e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") " pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.761305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.761274    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:09:54.891392 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:54.891363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:09:54.894485 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:09:54.894456    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod326f0a70_e7d9_4745_8c2b_9541f955f379.slice/crio-fc2a2d289a1ab669ecc4da465239d45aba60f8bc54043e285ed144c79d9b0675 WatchSource:0}: Error finding container fc2a2d289a1ab669ecc4da465239d45aba60f8bc54043e285ed144c79d9b0675: Status 404 returned error can't find the container with id fc2a2d289a1ab669ecc4da465239d45aba60f8bc54043e285ed144c79d9b0675
Jul 30 04:09:55.665636 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.665559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:55.666037 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.665826    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:55.678146 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.678124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:55.716492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.716461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" event={"ID":"326f0a70-e7d9-4745-8c2b-9541f955f379","Type":"ContainerStarted","Data":"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"}
Jul 30 04:09:55.716637 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.716502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" event={"ID":"326f0a70-e7d9-4745-8c2b-9541f955f379","Type":"ContainerStarted","Data":"fc2a2d289a1ab669ecc4da465239d45aba60f8bc54043e285ed144c79d9b0675"}
Jul 30 04:09:55.727802 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.727780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:09:55.733063 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:09:55.733026    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" podStartSLOduration=1.733014126 podStartE2EDuration="1.733014126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:09:55.731541585 +0000 UTC m=+1488.548478065" watchObservedRunningTime="2026-07-30 04:09:55.733014126 +0000 UTC m=+1488.549950602"
Jul 30 04:10:04.762353 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:04.762318    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:04.762773 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:04.762595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:04.774991 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:04.774963    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:05.150461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:05.150383    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:10:05.777051 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:05.777021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:14.784680 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:14.784646    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:10:15.901516 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:15.901481    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:10:19.453413 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:19.453376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:10:29.805481 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:29.805369    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" podUID="57f84b74-799a-4d6a-bd27-659caa1175d2" containerName="main" containerID="cri-o://711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca" gracePeriod=45
Jul 30 04:10:30.069819 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.069746    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:10:30.190381 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190347    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190381 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190442    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95hws\" (UniqueName: \"kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190564    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir\") pod \"57f84b74-799a-4d6a-bd27-659caa1175d2\" (UID: \"57f84b74-799a-4d6a-bd27-659caa1175d2\") "
Jul 30 04:10:30.190878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190667    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:30.190878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home" (OuterVolumeSpecName: "home") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:30.190878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190816    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.190878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190833    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.191068 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.190919    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:30.192736 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.192713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm" (OuterVolumeSpecName: "dshm") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:30.192867 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.192825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:10:30.192867 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.192852    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws" (OuterVolumeSpecName: "kube-api-access-95hws") pod "57f84b74-799a-4d6a-bd27-659caa1175d2" (UID: "57f84b74-799a-4d6a-bd27-659caa1175d2"). InnerVolumeSpecName "kube-api-access-95hws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:10:30.291761 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.291723    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.291761 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.291759    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f84b74-799a-4d6a-bd27-659caa1175d2-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.292022 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.291772    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f84b74-799a-4d6a-bd27-659caa1175d2-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.292022 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.291786    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hws\" (UniqueName: \"kubernetes.io/projected/57f84b74-799a-4d6a-bd27-659caa1175d2-kube-api-access-95hws\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:30.896939 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.896903    2562 generic.go:356] "Generic (PLEG): container finished" podID="57f84b74-799a-4d6a-bd27-659caa1175d2" containerID="711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca" exitCode=0
Jul 30 04:10:30.897426 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.896961    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"
Jul 30 04:10:30.897426 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.896993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" event={"ID":"57f84b74-799a-4d6a-bd27-659caa1175d2","Type":"ContainerDied","Data":"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"}
Jul 30 04:10:30.897426 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.897038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr" event={"ID":"57f84b74-799a-4d6a-bd27-659caa1175d2","Type":"ContainerDied","Data":"15a13d05fb5ae65167ddfcc17bb51ccbdf35870b9579a7427ab33c24e0cf615d"}
Jul 30 04:10:30.897426 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.897061    2562 scope.go:117] "RemoveContainer" containerID="711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"
Jul 30 04:10:30.906679 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.906659    2562 scope.go:117] "RemoveContainer" containerID="711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"
Jul 30 04:10:30.906988 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:10:30.906968    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca\": container with ID starting with 711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca not found: ID does not exist" containerID="711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"
Jul 30 04:10:30.907057 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.907001    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca"} err="failed to get container status \"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca\": rpc error: code = NotFound desc = could not find container \"711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca\": container with ID starting with 711a2abdf5f6ed560225822fa139d01aade4387fce9c12378b339645c7980eca not found: ID does not exist"
Jul 30 04:10:30.918415 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.918387    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:10:30.920981 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.920925    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" podUID="326f0a70-e7d9-4745-8c2b-9541f955f379" containerName="main" containerID="cri-o://49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf" gracePeriod=45
Jul 30 04:10:30.921994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:30.921969    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a3a71fd3-kserve-74897f7967-q4pbr"]
Jul 30 04:10:31.182950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.182924    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:31.299725 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299687    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.299916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.299916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.299916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299819    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.299916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299878    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg74p\" (UniqueName: \"kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.299916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.299911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs\") pod \"326f0a70-e7d9-4745-8c2b-9541f955f379\" (UID: \"326f0a70-e7d9-4745-8c2b-9541f955f379\") "
Jul 30 04:10:31.300168 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.300004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:31.300168 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.300044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache" (OuterVolumeSpecName: "model-cache") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:31.300277 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.300222    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.300277 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.300240    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.300488 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.300460    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home" (OuterVolumeSpecName: "home") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:31.302052 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.302028    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm" (OuterVolumeSpecName: "dshm") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:10:31.302052 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.302046    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p" (OuterVolumeSpecName: "kube-api-access-hg74p") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "kube-api-access-hg74p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:10:31.302217 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.302089    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "326f0a70-e7d9-4745-8c2b-9541f955f379" (UID: "326f0a70-e7d9-4745-8c2b-9541f955f379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:10:31.401672 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.401633    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.401672 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.401666    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326f0a70-e7d9-4745-8c2b-9541f955f379-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.401672 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.401680    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg74p\" (UniqueName: \"kubernetes.io/projected/326f0a70-e7d9-4745-8c2b-9541f955f379-kube-api-access-hg74p\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.401910 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.401694    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326f0a70-e7d9-4745-8c2b-9541f955f379-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:10:31.785478 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.785441    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f84b74-799a-4d6a-bd27-659caa1175d2" path="/var/lib/kubelet/pods/57f84b74-799a-4d6a-bd27-659caa1175d2/volumes"
Jul 30 04:10:31.902898 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.902860    2562 generic.go:356] "Generic (PLEG): container finished" podID="326f0a70-e7d9-4745-8c2b-9541f955f379" containerID="49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf" exitCode=0
Jul 30 04:10:31.902898 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.902898    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"
Jul 30 04:10:31.903411 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.902943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" event={"ID":"326f0a70-e7d9-4745-8c2b-9541f955f379","Type":"ContainerDied","Data":"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"}
Jul 30 04:10:31.903411 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.902984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m" event={"ID":"326f0a70-e7d9-4745-8c2b-9541f955f379","Type":"ContainerDied","Data":"fc2a2d289a1ab669ecc4da465239d45aba60f8bc54043e285ed144c79d9b0675"}
Jul 30 04:10:31.903411 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.903006    2562 scope.go:117] "RemoveContainer" containerID="49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"
Jul 30 04:10:31.912116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.912095    2562 scope.go:117] "RemoveContainer" containerID="49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"
Jul 30 04:10:31.912433 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:10:31.912407    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf\": container with ID starting with 49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf not found: ID does not exist" containerID="49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"
Jul 30 04:10:31.912542 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.912443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf"} err="failed to get container status \"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf\": rpc error: code = NotFound desc = could not find container \"49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf\": container with ID starting with 49f5dd92db26ca179c9ae64597a9cd87961a972feaa8f2417894060b7e63acbf not found: ID does not exist"
Jul 30 04:10:31.917921 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.917893    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:10:31.921694 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:31.921670    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a3a71fd3-kserve-86679ffd95-nvd5m"]
Jul 30 04:10:33.643785 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:33.643746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:10:33.784531 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:33.784485    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326f0a70-e7d9-4745-8c2b-9541f955f379" path="/var/lib/kubelet/pods/326f0a70-e7d9-4745-8c2b-9541f955f379/volumes"
Jul 30 04:10:40.353129 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:40.353088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:10:48.552906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:48.552866    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:10:55.449907 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:10:55.449868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:11:16.046119 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:11:16.046083    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:11:23.838768 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:11:23.838728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:11:35.751990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:11:35.751952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:11:45.053503 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:11:45.053466    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:11:54.840350 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:11:54.840313    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:07.245149 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:07.245106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:16.745217 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:16.745176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:26.549409 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:26.549373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:35.742361 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:35.742320    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:49.247938 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:49.247897    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:12:57.044189 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:12:57.044155    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:13:20.044256 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:13:20.044222    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:13:33.344794 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:13:33.344712    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:13:47.841316 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:13:47.841279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:14:03.044444 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:14:03.044406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:14:11.841382 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:14:11.841346    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:14:24.746937 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:14:24.746901    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:14:40.145490 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:14:40.145457    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:14:50.745376 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:14:50.745342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:15:00.448760 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:15:00.448676    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:15:08.646640 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:15:08.646601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:15:37.061194 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:15:37.061144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:15:46.447583 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:15:46.447541    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:16:11.340617 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:16:11.340583    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:16:21.140858 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:16:21.140820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:16:45.947093 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:16:45.947005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:16:56.142634 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:16:56.142600    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:17:20.355630 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:17:20.355588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:17:43.549386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:17:43.549350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:19:13.150558 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:19:13.150500    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:19:25.849395 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:19:25.849352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:19:37.447561 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:19:37.447511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:20:13.347445 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:20:13.347403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:20:52.243452 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:20:52.243413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:21:08.209768 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.209731    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:21:08.210346 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.209983    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" podUID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" containerName="manager" containerID="cri-o://d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6" gracePeriod=10
Jul 30 04:21:08.757414 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.757388    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:21:08.892912 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.892810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hgwn\" (UniqueName: \"kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn\") pod \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") "
Jul 30 04:21:08.893084 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.892957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume\") pod \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\" (UID: \"b105999e-78b2-4a14-b0c4-fab7e7caca4b\") "
Jul 30 04:21:08.893306 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.893284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b105999e-78b2-4a14-b0c4-fab7e7caca4b" (UID: "b105999e-78b2-4a14-b0c4-fab7e7caca4b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:21:08.894742 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.894719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn" (OuterVolumeSpecName: "kube-api-access-2hgwn") pod "b105999e-78b2-4a14-b0c4-fab7e7caca4b" (UID: "b105999e-78b2-4a14-b0c4-fab7e7caca4b"). InnerVolumeSpecName "kube-api-access-2hgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:21:08.939207 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.939170    2562 generic.go:356] "Generic (PLEG): container finished" podID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" containerID="d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6" exitCode=0
Jul 30 04:21:08.939330 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.939228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"
Jul 30 04:21:08.939330 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.939243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" event={"ID":"b105999e-78b2-4a14-b0c4-fab7e7caca4b","Type":"ContainerDied","Data":"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"}
Jul 30 04:21:08.939330 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.939284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv" event={"ID":"b105999e-78b2-4a14-b0c4-fab7e7caca4b","Type":"ContainerDied","Data":"1eed2e2b68df2ad2912d8fb98877b4f3075fadc6afea258e095bf17569296c61"}
Jul 30 04:21:08.939330 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.939305    2562 scope.go:117] "RemoveContainer" containerID="d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"
Jul 30 04:21:08.949370 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.949355    2562 scope.go:117] "RemoveContainer" containerID="d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"
Jul 30 04:21:08.949632 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:21:08.949614    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6\": container with ID starting with d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6 not found: ID does not exist" containerID="d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"
Jul 30 04:21:08.949686 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.949640    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6"} err="failed to get container status \"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6\": rpc error: code = NotFound desc = could not find container \"d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6\": container with ID starting with d730af51b608921e8f56b564d73c14a95c0e40fc162f0eca09e2b06904f466c6 not found: ID does not exist"
Jul 30 04:21:08.960382 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.960362    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:21:08.965607 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.965583    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9s8hv"]
Jul 30 04:21:08.994291 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.994273    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b105999e-78b2-4a14-b0c4-fab7e7caca4b-extensions-socket-volume\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:21:08.994291 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:08.994292    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hgwn\" (UniqueName: \"kubernetes.io/projected/b105999e-78b2-4a14-b0c4-fab7e7caca4b-kube-api-access-2hgwn\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:21:09.785101 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:21:09.785057    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" path="/var/lib/kubelet/pods/b105999e-78b2-4a14-b0c4-fab7e7caca4b/volumes"
Jul 30 04:22:23.851873 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.851835    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"]
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852156    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326f0a70-e7d9-4745-8c2b-9541f955f379" containerName="main"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852166    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="326f0a70-e7d9-4745-8c2b-9541f955f379" containerName="main"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" containerName="manager"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852193    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" containerName="manager"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852200    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f84b74-799a-4d6a-bd27-659caa1175d2" containerName="main"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f84b74-799a-4d6a-bd27-659caa1175d2" containerName="main"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852258    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f84b74-799a-4d6a-bd27-659caa1175d2" containerName="main"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852266    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b105999e-78b2-4a14-b0c4-fab7e7caca4b" containerName="manager"
Jul 30 04:22:23.852358 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.852273    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="326f0a70-e7d9-4745-8c2b-9541f955f379" containerName="main"
Jul 30 04:22:23.855432 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.855412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:23.858271 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.858250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhdjg\""
Jul 30 04:22:23.866201 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.866176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"]
Jul 30 04:22:23.978681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.978655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfz6\" (UniqueName: \"kubernetes.io/projected/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-kube-api-access-pbfz6\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:23.978813 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:23.978693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.079107 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.079074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfz6\" (UniqueName: \"kubernetes.io/projected/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-kube-api-access-pbfz6\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.079227 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.079111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.079436 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.079421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.087871 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.087853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfz6\" (UniqueName: \"kubernetes.io/projected/1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8-kube-api-access-pbfz6\") pod \"kuadrant-operator-controller-manager-86878c769f-n59hc\" (UID: \"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.165894 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.165827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:24.298913 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:24.298887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"]
Jul 30 04:22:24.300750 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:22:24.300722    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cff8f3e_dc2a_4bff_90ba_dd18404bcbd8.slice/crio-396d8af2d0d2cc2f9945b7e2b7a4fef0181e110502bdfa20312d46c780c39d6b WatchSource:0}: Error finding container 396d8af2d0d2cc2f9945b7e2b7a4fef0181e110502bdfa20312d46c780c39d6b: Status 404 returned error can't find the container with id 396d8af2d0d2cc2f9945b7e2b7a4fef0181e110502bdfa20312d46c780c39d6b
Jul 30 04:22:25.283681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:25.283644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc" event={"ID":"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8","Type":"ContainerStarted","Data":"9e2fa03f70e8dd457b81461e4f2d5bc42ead81781fcfea62a87d88b8c240644b"}
Jul 30 04:22:25.284043 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:25.283698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc" event={"ID":"1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8","Type":"ContainerStarted","Data":"396d8af2d0d2cc2f9945b7e2b7a4fef0181e110502bdfa20312d46c780c39d6b"}
Jul 30 04:22:25.284043 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:25.283758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:25.306349 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:25.306297    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc" podStartSLOduration=2.306285339 podStartE2EDuration="2.306285339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:22:25.304702861 +0000 UTC m=+2238.121639340" watchObservedRunningTime="2026-07-30 04:22:25.306285339 +0000 UTC m=+2238.123221815"
Jul 30 04:22:36.290400 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:36.290368    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n59hc"
Jul 30 04:22:46.946840 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:46.946805    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:22:58.743067 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:22:58.743028    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:23:20.347743 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.347704    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:23:20.351600 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.351581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.361506 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.361468    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bd1da342\"/\"openshift-service-ca.crt\""
Jul 30 04:23:20.361693 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.361580    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bd1da342\"/\"e2e-lifecycle-model-bd1da342-kserve-self-signed-certs\""
Jul 30 04:23:20.361693 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.361617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bd1da342\"/\"default-dockercfg-9qqbn\""
Jul 30 04:23:20.361879 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.361854    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bd1da342\"/\"kube-root-ca.crt\""
Jul 30 04:23:20.365340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.365291    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:23:20.433548 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.433733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.433733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.433733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvg8h\" (UniqueName: \"kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.433733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.433733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.433684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534240 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvg8h\" (UniqueName: \"kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534766 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534822 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.534874 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.534852    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.536828 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.536802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.537249 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.537232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.542586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.542566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvg8h\" (UniqueName: \"kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h\") pod \"e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") " pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.668491 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.668416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:20.805344 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:20.805315    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:23:20.806996 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:23:20.806969    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd72176_6e6d_4d54_9bf2_05b8148b332d.slice/crio-5ad463e053974ef84aab35f879f85270d143e1dfa5257204c0fee2ebe1c0fd58 WatchSource:0}: Error finding container 5ad463e053974ef84aab35f879f85270d143e1dfa5257204c0fee2ebe1c0fd58: Status 404 returned error can't find the container with id 5ad463e053974ef84aab35f879f85270d143e1dfa5257204c0fee2ebe1c0fd58
Jul 30 04:23:21.545212 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:21.545179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" event={"ID":"7fd72176-6e6d-4d54-9bf2-05b8148b332d","Type":"ContainerStarted","Data":"5f2a0d9eeb66a841e56787b4c0ca4bafea8a0ea542368c47dd71fa047c3c39fa"}
Jul 30 04:23:21.545582 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:21.545227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" event={"ID":"7fd72176-6e6d-4d54-9bf2-05b8148b332d","Type":"ContainerStarted","Data":"5ad463e053974ef84aab35f879f85270d143e1dfa5257204c0fee2ebe1c0fd58"}
Jul 30 04:23:21.562250 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:21.562207    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" podStartSLOduration=1.5621903160000001 podStartE2EDuration="1.562190316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:23:21.561299708 +0000 UTC m=+2294.378236186" watchObservedRunningTime="2026-07-30 04:23:21.562190316 +0000 UTC m=+2294.379126792"
Jul 30 04:23:23.645921 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:23.645887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:23:30.669232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:30.669195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:30.669232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:30.669240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:30.681625 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:30.681603    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:31.606662 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:31.606629    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:23:42.746641 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:23:42.746603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:24:09.848041 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:09.848003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:24:19.710530 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:19.710478    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:24:20.559927 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:20.559888    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:24:34.729419 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:34.729369    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" podUID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" containerName="main" containerID="cri-o://5f2a0d9eeb66a841e56787b4c0ca4bafea8a0ea542368c47dd71fa047c3c39fa" gracePeriod=45
Jul 30 04:24:34.889686 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:34.889653    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" containerID="5f2a0d9eeb66a841e56787b4c0ca4bafea8a0ea542368c47dd71fa047c3c39fa" exitCode=0
Jul 30 04:24:34.889855 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:34.889722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" event={"ID":"7fd72176-6e6d-4d54-9bf2-05b8148b332d","Type":"ContainerDied","Data":"5f2a0d9eeb66a841e56787b4c0ca4bafea8a0ea542368c47dd71fa047c3c39fa"}
Jul 30 04:24:34.991021 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:34.990961    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:24:35.156360 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156385    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156482    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvg8h\" (UniqueName: \"kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156558    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm\") pod \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\" (UID: \"7fd72176-6e6d-4d54-9bf2-05b8148b332d\") "
Jul 30 04:24:35.156774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156637    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:24:35.156774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156678    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home" (OuterVolumeSpecName: "home") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:24:35.156927 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156799    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:24:35.156927 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156828    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.156927 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.156847    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.158863 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.158827    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm" (OuterVolumeSpecName: "dshm") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:24:35.158863 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.158845    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h" (OuterVolumeSpecName: "kube-api-access-zvg8h") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "kube-api-access-zvg8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:24:35.159006 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.158912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fd72176-6e6d-4d54-9bf2-05b8148b332d" (UID: "7fd72176-6e6d-4d54-9bf2-05b8148b332d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:24:35.257842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.257752    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd72176-6e6d-4d54-9bf2-05b8148b332d-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.257842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.257782    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvg8h\" (UniqueName: \"kubernetes.io/projected/7fd72176-6e6d-4d54-9bf2-05b8148b332d-kube-api-access-zvg8h\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.257842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.257791    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.257842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.257800    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd72176-6e6d-4d54-9bf2-05b8148b332d-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:24:35.895752 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.895719    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"
Jul 30 04:24:35.896235 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.895723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp" event={"ID":"7fd72176-6e6d-4d54-9bf2-05b8148b332d","Type":"ContainerDied","Data":"5ad463e053974ef84aab35f879f85270d143e1dfa5257204c0fee2ebe1c0fd58"}
Jul 30 04:24:35.896235 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.895851    2562 scope.go:117] "RemoveContainer" containerID="5f2a0d9eeb66a841e56787b4c0ca4bafea8a0ea542368c47dd71fa047c3c39fa"
Jul 30 04:24:35.911809 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.911785    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:24:35.917083 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:35.917061    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bd1da342/e2e-lifecycle-model-bd1da342-kserve-b467b5d75-xhpvp"]
Jul 30 04:24:37.785163 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:24:37.785129    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" path="/var/lib/kubelet/pods/7fd72176-6e6d-4d54-9bf2-05b8148b332d/volumes"
Jul 30 04:25:05.146871 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:25:05.146831    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:25:28.746744 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:25:28.746703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:25:38.349130 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:25:38.349085    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:26:26.544873 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:26:26.544828    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:26:55.540538 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:26:55.540486    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:27:28.145884 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.145819    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:27:28.146492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.146351    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" containerName="main"
Jul 30 04:27:28.146492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.146366    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" containerName="main"
Jul 30 04:27:28.146492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.146464    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd72176-6e6d-4d54-9bf2-05b8148b332d" containerName="main"
Jul 30 04:27:28.149611 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.149589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.152142 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.152116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9acc4f\"/\"kube-root-ca.crt\""
Jul 30 04:27:28.152250 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.152203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9acc4f\"/\"openshift-service-ca.crt\""
Jul 30 04:27:28.153348 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.153333    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9acc4f\"/\"default-dockercfg-tzg7z\""
Jul 30 04:27:28.153426 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.153405    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9acc4f\"/\"test-model-9acc4f-kserve-self-signed-certs\""
Jul 30 04:27:28.157355 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.157330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:27:28.303619 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.303808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdx5\" (UniqueName: \"kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.303808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.303808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.303808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.303981 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.303818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404480 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404480 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404480 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdx5\" (UniqueName: \"kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404797 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404797 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.404797 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.404628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.405081 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.405059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.405161 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.405089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.405161 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.405127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.406885 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.406852    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.407037 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.407021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.412158 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.412138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdx5\" (UniqueName: \"kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5\") pod \"test-model-9acc4f-kserve-58fbf6498f-wb52n\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") " pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.461135 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.461101    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:28.594533 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.594490    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:27:28.596629 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:27:28.596596    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2351b697_8b8f_4a6a_882a_81f623f89d97.slice/crio-7486334e522f700831ecbec8f5ed59196e33d4eca39dcb0c6afefa2885de55e9 WatchSource:0}: Error finding container 7486334e522f700831ecbec8f5ed59196e33d4eca39dcb0c6afefa2885de55e9: Status 404 returned error can't find the container with id 7486334e522f700831ecbec8f5ed59196e33d4eca39dcb0c6afefa2885de55e9
Jul 30 04:27:28.696167 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.696090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" event={"ID":"2351b697-8b8f-4a6a-882a-81f623f89d97","Type":"ContainerStarted","Data":"594dd6fabce671a6e371aac8ec7be518328349646eee66bef7c003b09b98294f"}
Jul 30 04:27:28.696167 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.696133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" event={"ID":"2351b697-8b8f-4a6a-882a-81f623f89d97","Type":"ContainerStarted","Data":"7486334e522f700831ecbec8f5ed59196e33d4eca39dcb0c6afefa2885de55e9"}
Jul 30 04:27:28.712073 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:28.712027    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" podStartSLOduration=0.712012652 podStartE2EDuration="712.012652ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:27:28.710366306 +0000 UTC m=+2541.527302785" watchObservedRunningTime="2026-07-30 04:27:28.712012652 +0000 UTC m=+2541.528949130"
Jul 30 04:27:32.442171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:32.442136    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:27:38.461369 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:38.461335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:38.461759 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:38.461387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:38.474025 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:38.474001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:38.752901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:38.752875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:27:48.339258 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.339225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:27:48.342728 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.342708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.345117 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.345097    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c24378\"/\"kube-root-ca.crt\""
Jul 30 04:27:48.346176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.346156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c24378\"/\"test-model-c24378-kserve-self-signed-certs\""
Jul 30 04:27:48.346285 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.346179    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c24378\"/\"default-dockercfg-5z7m7\""
Jul 30 04:27:48.346285 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.346196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c24378\"/\"openshift-service-ca.crt\""
Jul 30 04:27:48.350745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.350654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:27:48.384981 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.384955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.385139 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.384995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.385139 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.385056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.385139 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.385095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.385266 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.385135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwg4\" (UniqueName: \"kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.385266 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.385205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486479 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwg4\" (UniqueName: \"kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.486971 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.486916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.488134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.488108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.488323 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.488150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.489298 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.489271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.489941 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.489912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.495328 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.495308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwg4\" (UniqueName: \"kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4\") pod \"test-model-c24378-kserve-fd66f897f-bfrkn\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") " pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.653359 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.653263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:48.792448 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:48.792419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:27:48.794177 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:27:48.794149    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b248c18_bea4_4707_aa0e_f8af05ac5b9d.slice/crio-64b961882c16b94848ff300ffa38fc6c397886e1e53dc57ac442c124bb7dd6fe WatchSource:0}: Error finding container 64b961882c16b94848ff300ffa38fc6c397886e1e53dc57ac442c124bb7dd6fe: Status 404 returned error can't find the container with id 64b961882c16b94848ff300ffa38fc6c397886e1e53dc57ac442c124bb7dd6fe
Jul 30 04:27:49.789694 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:49.789650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" event={"ID":"0b248c18-bea4-4707-aa0e-f8af05ac5b9d","Type":"ContainerStarted","Data":"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"}
Jul 30 04:27:49.790165 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:49.789705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" event={"ID":"0b248c18-bea4-4707-aa0e-f8af05ac5b9d","Type":"ContainerStarted","Data":"64b961882c16b94848ff300ffa38fc6c397886e1e53dc57ac442c124bb7dd6fe"}
Jul 30 04:27:49.807645 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:49.807600    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" podStartSLOduration=1.807585024 podStartE2EDuration="1.807585024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:27:49.806382872 +0000 UTC m=+2562.623319363" watchObservedRunningTime="2026-07-30 04:27:49.807585024 +0000 UTC m=+2562.624521501"
Jul 30 04:27:53.542747 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:53.542697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:27:58.653596 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:58.653559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:58.654120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:58.653620    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:58.666541 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:58.666503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:27:58.840622 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:27:58.840593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:28:27.792844 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:27.792808    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:28:28.683670 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:28.683639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:28:31.342048 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:31.341960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:28:42.812862 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:42.812787    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" podUID="2351b697-8b8f-4a6a-882a-81f623f89d97" containerName="main" containerID="cri-o://594dd6fabce671a6e371aac8ec7be518328349646eee66bef7c003b09b98294f" gracePeriod=45
Jul 30 04:28:43.039827 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.039783    2562 generic.go:356] "Generic (PLEG): container finished" podID="2351b697-8b8f-4a6a-882a-81f623f89d97" containerID="594dd6fabce671a6e371aac8ec7be518328349646eee66bef7c003b09b98294f" exitCode=0
Jul 30 04:28:43.039990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.039824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" event={"ID":"2351b697-8b8f-4a6a-882a-81f623f89d97","Type":"ContainerDied","Data":"594dd6fabce671a6e371aac8ec7be518328349646eee66bef7c003b09b98294f"}
Jul 30 04:28:43.070292 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.070239    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:28:43.144950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.144913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.144950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.144952    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.145205 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.144981    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvdx5\" (UniqueName: \"kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.145205 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.145205 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.145205 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145148    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm\") pod \"2351b697-8b8f-4a6a-882a-81f623f89d97\" (UID: \"2351b697-8b8f-4a6a-882a-81f623f89d97\") "
Jul 30 04:28:43.145461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145247    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:43.145461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home" (OuterVolumeSpecName: "home") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:43.145461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache" (OuterVolumeSpecName: "model-cache") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:43.145595 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145474    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.145595 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145486    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.145595 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.145495    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.147221 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.147198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:28:43.147312 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.147243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5" (OuterVolumeSpecName: "kube-api-access-lvdx5") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "kube-api-access-lvdx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:28:43.147312 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.147298    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm" (OuterVolumeSpecName: "dshm") pod "2351b697-8b8f-4a6a-882a-81f623f89d97" (UID: "2351b697-8b8f-4a6a-882a-81f623f89d97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:43.246857 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.246814    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2351b697-8b8f-4a6a-882a-81f623f89d97-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.246857 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.246846    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2351b697-8b8f-4a6a-882a-81f623f89d97-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.246857 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.246857    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdx5\" (UniqueName: \"kubernetes.io/projected/2351b697-8b8f-4a6a-882a-81f623f89d97-kube-api-access-lvdx5\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:43.703909 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.703868    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" podUID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" containerName="main" containerID="cri-o://10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24" gracePeriod=45
Jul 30 04:28:43.952953 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.952922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:28:43.953496 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:43.953477    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:28:44.046731 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.046697    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" containerID="10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24" exitCode=0
Jul 30 04:28:44.046924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.046743    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"
Jul 30 04:28:44.046924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.046782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" event={"ID":"0b248c18-bea4-4707-aa0e-f8af05ac5b9d","Type":"ContainerDied","Data":"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"}
Jul 30 04:28:44.046924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.046833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn" event={"ID":"0b248c18-bea4-4707-aa0e-f8af05ac5b9d","Type":"ContainerDied","Data":"64b961882c16b94848ff300ffa38fc6c397886e1e53dc57ac442c124bb7dd6fe"}
Jul 30 04:28:44.046924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.046868    2562 scope.go:117] "RemoveContainer" containerID="10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"
Jul 30 04:28:44.048374 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.048354    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"
Jul 30 04:28:44.048374 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.048363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n" event={"ID":"2351b697-8b8f-4a6a-882a-81f623f89d97","Type":"ContainerDied","Data":"7486334e522f700831ecbec8f5ed59196e33d4eca39dcb0c6afefa2885de55e9"}
Jul 30 04:28:44.055324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.055456 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055370    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qwg4\" (UniqueName: \"kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.055456 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.055720 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055456    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.055720 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055567    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:44.055720 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055699    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.055720 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:44.055946 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home" (OuterVolumeSpecName: "home") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:44.055946 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.055771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm\") pod \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\" (UID: \"0b248c18-bea4-4707-aa0e-f8af05ac5b9d\") "
Jul 30 04:28:44.056120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056065    2562 scope.go:117] "RemoveContainer" containerID="10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"
Jul 30 04:28:44.056120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056099    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.056120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056118    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.056271 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056132    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.056414 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:28:44.056386    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24\": container with ID starting with 10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24 not found: ID does not exist" containerID="10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"
Jul 30 04:28:44.056551 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056423    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24"} err="failed to get container status \"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24\": rpc error: code = NotFound desc = could not find container \"10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24\": container with ID starting with 10343a98d8fce3d13cbcdc99a100cc4e27a1105cc60609c6ab9b8f9155162f24 not found: ID does not exist"
Jul 30 04:28:44.056551 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.056447    2562 scope.go:117] "RemoveContainer" containerID="594dd6fabce671a6e371aac8ec7be518328349646eee66bef7c003b09b98294f"
Jul 30 04:28:44.058106 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.058078    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:28:44.058106 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.058093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm" (OuterVolumeSpecName: "dshm") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:28:44.058243 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.058078    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4" (OuterVolumeSpecName: "kube-api-access-6qwg4") pod "0b248c18-bea4-4707-aa0e-f8af05ac5b9d" (UID: "0b248c18-bea4-4707-aa0e-f8af05ac5b9d"). InnerVolumeSpecName "kube-api-access-6qwg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:28:44.065580 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.065555    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:28:44.067290 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.067269    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9acc4f/test-model-9acc4f-kserve-58fbf6498f-wb52n"]
Jul 30 04:28:44.156715 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.156683    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwg4\" (UniqueName: \"kubernetes.io/projected/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-kube-api-access-6qwg4\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.156715 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.156710    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.156715 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.156723    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b248c18-bea4-4707-aa0e-f8af05ac5b9d-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:28:44.368308 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.368276    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:28:44.369749 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:44.369731    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c24378/test-model-c24378-kserve-fd66f897f-bfrkn"]
Jul 30 04:28:45.784836 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:45.784800    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" path="/var/lib/kubelet/pods/0b248c18-bea4-4707-aa0e-f8af05ac5b9d/volumes"
Jul 30 04:28:45.785214 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:28:45.785193    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2351b697-8b8f-4a6a-882a-81f623f89d97" path="/var/lib/kubelet/pods/2351b697-8b8f-4a6a-882a-81f623f89d97/volumes"
Jul 30 04:29:00.236121 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:29:00.236090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:29:11.945547 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:29:11.945500    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:29:42.938002 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:29:42.937965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:30:00.135511 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135431    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"]
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135777    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2351b697-8b8f-4a6a-882a-81f623f89d97" containerName="main"
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135787    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2351b697-8b8f-4a6a-882a-81f623f89d97" containerName="main"
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135805    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" containerName="main"
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135811    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" containerName="main"
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2351b697-8b8f-4a6a-882a-81f623f89d97" containerName="main"
Jul 30 04:30:00.135881 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.135883    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b248c18-bea4-4707-aa0e-f8af05ac5b9d" containerName="main"
Jul 30 04:30:00.139049 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.139022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"
Jul 30 04:30:00.139207 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.139185    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"]
Jul 30 04:30:00.141501 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.141479    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:30:00.142381 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.142362    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"]
Jul 30 04:30:00.142501 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.142476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"
Jul 30 04:30:00.146458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.146434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"]
Jul 30 04:30:00.146594 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.146560    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"
Jul 30 04:30:00.151764 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.151738    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"]
Jul 30 04:30:00.154014 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.153991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"]
Jul 30 04:30:00.154949 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.154932    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"
Jul 30 04:30:00.162967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.162943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"
Jul 30 04:30:00.198686 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.198660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"
Jul 30 04:30:00.321096 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.320627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"]
Jul 30 04:30:00.323038 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:30:00.323003    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf564bb23_6373_4222_bea2_423beedf3fb3.slice/crio-805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f WatchSource:0}: Error finding container 805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f: Status 404 returned error can't find the container with id 805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f
Jul 30 04:30:00.325391 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.325362    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:30:00.371386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.371361    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"]
Jul 30 04:30:00.372946 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:30:00.372918    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10017c9_58b8_4943_860c_65fc8b7f403a.slice/crio-bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15 WatchSource:0}: Error finding container bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15: Status 404 returned error can't find the container with id bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15
Jul 30 04:30:00.402857 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.402782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc" event={"ID":"f10017c9-58b8-4943-860c-65fc8b7f403a","Type":"ContainerStarted","Data":"bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15"}
Jul 30 04:30:00.403895 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.403875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22" event={"ID":"f564bb23-6373-4222-bea2-423beedf3fb3","Type":"ContainerStarted","Data":"805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f"}
Jul 30 04:30:00.549056 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:00.549029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"]
Jul 30 04:30:00.551317 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:30:00.551292    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdaa638_fd00_485e_a88e_5e123285fb67.slice/crio-183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f WatchSource:0}: Error finding container 183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f: Status 404 returned error can't find the container with id 183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f
Jul 30 04:30:01.410054 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:01.410011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct" event={"ID":"afdaa638-fd00-485e-a88e-5e123285fb67","Type":"ContainerStarted","Data":"183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f"}
Jul 30 04:30:03.423774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:03.423744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc" event={"ID":"f10017c9-58b8-4943-860c-65fc8b7f403a","Type":"ContainerStarted","Data":"ed76dbcfc9a077c90e07c89ebcaae0f10e0e5cd2abd95c0cdeec3bc0875849d4"}
Jul 30 04:30:03.425203 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:03.425176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22" event={"ID":"f564bb23-6373-4222-bea2-423beedf3fb3","Type":"ContainerStarted","Data":"78d5af8906ea73a6e6ca2d3bcda676775519d37ab0a15d07073b75eeada5f2c9"}
Jul 30 04:30:03.440939 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:03.440891    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc" podStartSLOduration=0.939387171 podStartE2EDuration="3.440874266s" totalImagesPullingTime="2.501487095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:03.43831596 +0000 UTC m=+2696.255252433" watchObservedRunningTime="2026-07-30 04:30:03.440874266 +0000 UTC m=+2696.257810744"
Jul 30 04:30:03.452878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:03.452840    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22" podStartSLOduration=0.908520784 podStartE2EDuration="3.452828954s" totalImagesPullingTime="2.54430817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:03.450884849 +0000 UTC m=+2696.267821339" watchObservedRunningTime="2026-07-30 04:30:03.452828954 +0000 UTC m=+2696.269765430"
Jul 30 04:30:04.437599 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.437558    2562 generic.go:356] "Generic (PLEG): container finished" podID="afdaa638-fd00-485e-a88e-5e123285fb67" containerID="8f73bf09cb03f4a00a51bb6b201a3d711c06a3724d8944b5f644ce3327f71794" exitCode=0
Jul 30 04:30:04.438010 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.437608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct" event={"ID":"afdaa638-fd00-485e-a88e-5e123285fb67","Type":"ContainerDied","Data":"8f73bf09cb03f4a00a51bb6b201a3d711c06a3724d8944b5f644ce3327f71794"}
Jul 30 04:30:04.439075 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.439050    2562 generic.go:356] "Generic (PLEG): container finished" podID="f10017c9-58b8-4943-860c-65fc8b7f403a" containerID="ed76dbcfc9a077c90e07c89ebcaae0f10e0e5cd2abd95c0cdeec3bc0875849d4" exitCode=0
Jul 30 04:30:04.439210 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.439131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc" event={"ID":"f10017c9-58b8-4943-860c-65fc8b7f403a","Type":"ContainerDied","Data":"ed76dbcfc9a077c90e07c89ebcaae0f10e0e5cd2abd95c0cdeec3bc0875849d4"}
Jul 30 04:30:04.440421 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.440401    2562 generic.go:356] "Generic (PLEG): container finished" podID="f564bb23-6373-4222-bea2-423beedf3fb3" containerID="78d5af8906ea73a6e6ca2d3bcda676775519d37ab0a15d07073b75eeada5f2c9" exitCode=0
Jul 30 04:30:04.440514 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:04.440426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22" event={"ID":"f564bb23-6373-4222-bea2-423beedf3fb3","Type":"ContainerDied","Data":"78d5af8906ea73a6e6ca2d3bcda676775519d37ab0a15d07073b75eeada5f2c9"}
Jul 30 04:30:05.212476 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.212436    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"]
Jul 30 04:30:05.215840 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.215823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:05.218173 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.218151    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 04:30:05.222789 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.222674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"]
Jul 30 04:30:05.318880 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.318848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x\" (UID: \"b113e272-f311-4585-a1ca-0e908ebe4629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:05.420142 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.420108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x\" (UID: \"b113e272-f311-4585-a1ca-0e908ebe4629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:05.420716 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.420698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x\" (UID: \"b113e272-f311-4585-a1ca-0e908ebe4629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:05.527827 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.527803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:05.629785 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.629514    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"
Jul 30 04:30:05.670484 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.670456    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"
Jul 30 04:30:05.676107 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.676084    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"
Jul 30 04:30:05.715023 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:05.715001    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"]
Jul 30 04:30:05.717128 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:30:05.717102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb113e272_f311_4585_a1ca_0e908ebe4629.slice/crio-9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86 WatchSource:0}: Error finding container 9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86: Status 404 returned error can't find the container with id 9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86
Jul 30 04:30:06.451304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.451211    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct"
Jul 30 04:30:06.451304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.451245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756430-m2xct" event={"ID":"afdaa638-fd00-485e-a88e-5e123285fb67","Type":"ContainerDied","Data":"183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f"}
Jul 30 04:30:06.451304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.451298    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="183dbce650c1910f5eb17a50b343532cd94555f72070e77c0838759e0bc52b3f"
Jul 30 04:30:06.452769 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.452743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x" event={"ID":"b113e272-f311-4585-a1ca-0e908ebe4629","Type":"ContainerStarted","Data":"d29f31fee941aee228296c135e543ac1eb40682eece58b5e9cbd72a7dc319060"}
Jul 30 04:30:06.452917 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.452796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x" event={"ID":"b113e272-f311-4585-a1ca-0e908ebe4629","Type":"ContainerStarted","Data":"9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86"}
Jul 30 04:30:06.454284 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.454266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"
Jul 30 04:30:06.454386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.454307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc" event={"ID":"f10017c9-58b8-4943-860c-65fc8b7f403a","Type":"ContainerDied","Data":"bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15"}
Jul 30 04:30:06.454386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.454349    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdbf25bbe18909df1d63959698f8a52d77de12f234c19c50dde3b69adc75cd15"
Jul 30 04:30:06.455497 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.455473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22" event={"ID":"f564bb23-6373-4222-bea2-423beedf3fb3","Type":"ContainerDied","Data":"805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f"}
Jul 30 04:30:06.455628 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.455505    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"
Jul 30 04:30:06.455628 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.455539    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805c2e18e3497a5025cb2a99e249188c5159b05cfc0abbff7b35b4135af50f5f"
Jul 30 04:30:06.468096 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:06.468058    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x" podStartSLOduration=1.468046373 podStartE2EDuration="1.468046373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:06.466212894 +0000 UTC m=+2699.283149368" watchObservedRunningTime="2026-07-30 04:30:06.468046373 +0000 UTC m=+2699.284982892"
Jul 30 04:30:07.462802 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:07.462770    2562 generic.go:356] "Generic (PLEG): container finished" podID="b113e272-f311-4585-a1ca-0e908ebe4629" containerID="d29f31fee941aee228296c135e543ac1eb40682eece58b5e9cbd72a7dc319060" exitCode=0
Jul 30 04:30:07.463181 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:07.462831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x" event={"ID":"b113e272-f311-4585-a1ca-0e908ebe4629","Type":"ContainerDied","Data":"d29f31fee941aee228296c135e543ac1eb40682eece58b5e9cbd72a7dc319060"}
Jul 30 04:30:08.605949 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:08.605919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:08.748669 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:08.748642    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca\") pod \"b113e272-f311-4585-a1ca-0e908ebe4629\" (UID: \"b113e272-f311-4585-a1ca-0e908ebe4629\") "
Jul 30 04:30:08.748965 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:08.748941    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b113e272-f311-4585-a1ca-0e908ebe4629" (UID: "b113e272-f311-4585-a1ca-0e908ebe4629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:30:08.850039 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:08.850011    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b113e272-f311-4585-a1ca-0e908ebe4629-maas-api-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:30:09.474316 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:09.474285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x" event={"ID":"b113e272-f311-4585-a1ca-0e908ebe4629","Type":"ContainerDied","Data":"9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86"}
Jul 30 04:30:09.474316 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:09.474308    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"
Jul 30 04:30:09.474532 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:09.474332    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bbb029f964615c16f16060aa022ba958b9a2cef601aee8d3fc24c7126b81f86"
Jul 30 04:30:20.839993 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:20.839951    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"]
Jul 30 04:30:20.842098 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:20.842074    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6c99-29756430-r7s22"]
Jul 30 04:30:21.013745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:21.013704    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"]
Jul 30 04:30:21.015645 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:21.015615    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6c99-5jp5x"]
Jul 30 04:30:21.784928 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:21.784897    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b113e272-f311-4585-a1ca-0e908ebe4629" path="/var/lib/kubelet/pods/b113e272-f311-4585-a1ca-0e908ebe4629/volumes"
Jul 30 04:30:21.785165 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:21.785152    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f564bb23-6373-4222-bea2-423beedf3fb3" path="/var/lib/kubelet/pods/f564bb23-6373-4222-bea2-423beedf3fb3/volumes"
Jul 30 04:30:23.141763 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:23.141725    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:30:27.571713 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.571679    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"]
Jul 30 04:30:27.572216 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572197    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10017c9-58b8-4943-860c-65fc8b7f403a" containerName="cleanup"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572220    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10017c9-58b8-4943-860c-65fc8b7f403a" containerName="cleanup"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572233    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afdaa638-fd00-485e-a88e-5e123285fb67" containerName="cleanup"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572241    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="afdaa638-fd00-485e-a88e-5e123285fb67" containerName="cleanup"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572252    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b113e272-f311-4585-a1ca-0e908ebe4629" containerName="revoke-keys"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572261    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b113e272-f311-4585-a1ca-0e908ebe4629" containerName="revoke-keys"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572276    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f564bb23-6373-4222-bea2-423beedf3fb3" containerName="cleanup"
Jul 30 04:30:27.572297 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572284    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f564bb23-6373-4222-bea2-423beedf3fb3" containerName="cleanup"
Jul 30 04:30:27.572678 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572388    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="afdaa638-fd00-485e-a88e-5e123285fb67" containerName="cleanup"
Jul 30 04:30:27.572678 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572399    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10017c9-58b8-4943-860c-65fc8b7f403a" containerName="cleanup"
Jul 30 04:30:27.572678 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572408    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f564bb23-6373-4222-bea2-423beedf3fb3" containerName="cleanup"
Jul 30 04:30:27.572678 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.572419    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b113e272-f311-4585-a1ca-0e908ebe4629" containerName="revoke-keys"
Jul 30 04:30:27.576509 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.576490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:27.579665 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.579643    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 04:30:27.579936 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.579709    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:30:27.580950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.580929    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"]
Jul 30 04:30:27.712846 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.712808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587da0-8mbl7\" (UID: \"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:27.813730 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.813693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587da0-8mbl7\" (UID: \"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:27.814342 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.814320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-587da0-8mbl7\" (UID: \"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:27.887745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:27.887654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:28.023513 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:28.023483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"]
Jul 30 04:30:28.027616 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:30:28.027584    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeb50b1_9e26_48d3_a249_87a5cdab4ab9.slice/crio-d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e WatchSource:0}: Error finding container d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e: Status 404 returned error can't find the container with id d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e
Jul 30 04:30:28.569276 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:28.569240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7" event={"ID":"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9","Type":"ContainerStarted","Data":"a1ff5d05a454ccb89d7b80abd364a90ed54aec0c5d28bc9bb25b6bff61f05465"}
Jul 30 04:30:28.569458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:28.569292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7" event={"ID":"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9","Type":"ContainerStarted","Data":"d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e"}
Jul 30 04:30:28.586391 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:28.586339    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7" podStartSLOduration=1.586323885 podStartE2EDuration="1.586323885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:30:28.584892133 +0000 UTC m=+2721.401828611" watchObservedRunningTime="2026-07-30 04:30:28.586323885 +0000 UTC m=+2721.403260391"
Jul 30 04:30:29.575340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:29.575303    2562 generic.go:356] "Generic (PLEG): container finished" podID="2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" containerID="a1ff5d05a454ccb89d7b80abd364a90ed54aec0c5d28bc9bb25b6bff61f05465" exitCode=0
Jul 30 04:30:29.575513 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:29.575356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7" event={"ID":"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9","Type":"ContainerDied","Data":"a1ff5d05a454ccb89d7b80abd364a90ed54aec0c5d28bc9bb25b6bff61f05465"}
Jul 30 04:30:30.715142 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:30.715114    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:30.740244 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:30.740213    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca\") pod \"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9\" (UID: \"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9\") "
Jul 30 04:30:30.740542 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:30.740510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" (UID: "2eeb50b1-9e26-48d3-a249-87a5cdab4ab9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:30:30.840928 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:30.840840    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9-maas-api-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:30:31.586314 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:31.586287    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"
Jul 30 04:30:31.586540 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:31.586286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7" event={"ID":"2eeb50b1-9e26-48d3-a249-87a5cdab4ab9","Type":"ContainerDied","Data":"d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e"}
Jul 30 04:30:31.586540 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:31.586423    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d647e17596be1796feae8cab557451543f4a3a326bd12ed55d6b0e456d726f9e"
Jul 30 04:30:43.172087 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.172048    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"]
Jul 30 04:30:43.173915 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.173888    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-587da0-29756430-5cqzc"]
Jul 30 04:30:43.339045 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.339012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"]
Jul 30 04:30:43.341344 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.341321    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-587da0-8mbl7"]
Jul 30 04:30:43.785030 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.784998    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" path="/var/lib/kubelet/pods/2eeb50b1-9e26-48d3-a249-87a5cdab4ab9/volumes"
Jul 30 04:30:43.785282 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:43.785268    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10017c9-58b8-4943-860c-65fc8b7f403a" path="/var/lib/kubelet/pods/f10017c9-58b8-4943-860c-65fc8b7f403a/volumes"
Jul 30 04:30:47.941907 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:47.941874    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:30:58.750046 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:30:58.750012    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:31:29.039596 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:29.039564    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:31:50.947367 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.947333    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:31:50.947789 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.947718    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" containerName="revoke-keys"
Jul 30 04:31:50.947789 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.947731    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" containerName="revoke-keys"
Jul 30 04:31:50.947789 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.947788    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eeb50b1-9e26-48d3-a249-87a5cdab4ab9" containerName="revoke-keys"
Jul 30 04:31:50.950785 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.950768    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:50.954294 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.954269    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"default-dockercfg-4ng5c\""
Jul 30 04:31:50.954447 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.954312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"openshift-service-ca.crt\""
Jul 30 04:31:50.954447 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.954347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"kube-root-ca.crt\""
Jul 30 04:31:50.954447 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.954269    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"auth-test-model-324a29-kserve-self-signed-certs\""
Jul 30 04:31:50.961090 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:50.961064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:31:51.076178 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r257\" (UniqueName: \"kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.076360 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.076360 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.076360 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.076360 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.076566 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.076381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177701 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177701 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177977 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r257\" (UniqueName: \"kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177977 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177977 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.177977 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.177824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.178169 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.178139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.178169 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.178152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.178237 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.178215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.180119 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.180089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.180216 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.180198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.185676 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.185658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r257\" (UniqueName: \"kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257\") pod \"auth-test-model-324a29-kserve-57b88bd7f5-9vgnj\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") " pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.260446 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.260413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:31:51.392417 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.392382    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:31:51.394861 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:31:51.394834    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6afa3ec8_1447_4059_b275_fd51253aee30.slice/crio-bd9ef1e82e56862e87d0719c739f93356127cb60ee00e2c5903222781599d4b3 WatchSource:0}: Error finding container bd9ef1e82e56862e87d0719c739f93356127cb60ee00e2c5903222781599d4b3: Status 404 returned error can't find the container with id bd9ef1e82e56862e87d0719c739f93356127cb60ee00e2c5903222781599d4b3
Jul 30 04:31:51.972323 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.972280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" event={"ID":"6afa3ec8-1447-4059-b275-fd51253aee30","Type":"ContainerStarted","Data":"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"}
Jul 30 04:31:51.972323 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.972327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" event={"ID":"6afa3ec8-1447-4059-b275-fd51253aee30","Type":"ContainerStarted","Data":"bd9ef1e82e56862e87d0719c739f93356127cb60ee00e2c5903222781599d4b3"}
Jul 30 04:31:51.988500 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:51.988450    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" podStartSLOduration=1.988433691 podStartE2EDuration="1.988433691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:31:51.987881703 +0000 UTC m=+2804.804818182" watchObservedRunningTime="2026-07-30 04:31:51.988433691 +0000 UTC m=+2804.805370169"
Jul 30 04:31:54.348410 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:31:54.348375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:32:01.261101 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:01.261060    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:32:01.263712 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:01.261125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:32:01.273768 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:01.273748    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:32:02.034387 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:02.034358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:32:09.249975 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.249940    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:32:09.253351 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.253330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.255859 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.255837    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"openshift-service-ca.crt\""
Jul 30 04:32:09.255992 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.255844    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"default-dockercfg-d52rx\""
Jul 30 04:32:09.256942 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.256924    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"auth-test-model-64af9f-kserve-self-signed-certs\""
Jul 30 04:32:09.256999 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.256964    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"kube-root-ca.crt\""
Jul 30 04:32:09.262874 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.262855    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:32:09.331815 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.331786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.331987 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.331828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.331987 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.331856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.331987 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.331969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.332099 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.332007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.332099 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.332084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44t7\" (UniqueName: \"kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433046 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44t7\" (UniqueName: \"kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433215 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433215 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433215 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433215 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433215 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433498 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433591 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433499    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.433646 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.433588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.435557 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.435515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.435795 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.435777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.442409 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.442385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44t7\" (UniqueName: \"kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7\") pod \"auth-test-model-64af9f-kserve-59694ff9f5-ktbs2\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") " pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.563326 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.563242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:09.697530 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:09.697494    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:32:09.699419 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:32:09.699388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13dadd0b_7c26_4549_83bc_cdea94bc92fe.slice/crio-1d4ab8c3c634dda4d7cb585f7c921642822d646a2e72448b87fb0d4a6945e498 WatchSource:0}: Error finding container 1d4ab8c3c634dda4d7cb585f7c921642822d646a2e72448b87fb0d4a6945e498: Status 404 returned error can't find the container with id 1d4ab8c3c634dda4d7cb585f7c921642822d646a2e72448b87fb0d4a6945e498
Jul 30 04:32:10.067387 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:10.067352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" event={"ID":"13dadd0b-7c26-4549-83bc-cdea94bc92fe","Type":"ContainerStarted","Data":"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"}
Jul 30 04:32:10.067387 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:10.067394    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" event={"ID":"13dadd0b-7c26-4549-83bc-cdea94bc92fe","Type":"ContainerStarted","Data":"1d4ab8c3c634dda4d7cb585f7c921642822d646a2e72448b87fb0d4a6945e498"}
Jul 30 04:32:10.084730 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:10.084682    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" podStartSLOduration=1.084665578 podStartE2EDuration="1.084665578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:32:10.083167551 +0000 UTC m=+2822.900104028" watchObservedRunningTime="2026-07-30 04:32:10.084665578 +0000 UTC m=+2822.901602055"
Jul 30 04:32:13.049642 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:13.049604    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:32:19.563513 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:19.563472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:19.563968 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:19.563556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:19.576060 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:19.576037    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:20.139198 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:20.139167    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:32:37.946537 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:32:37.946496    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:33:03.750902 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:33:03.750868    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:33:21.442034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:33:21.441999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:33:47.442450 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:33:47.442413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:34:04.938925 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:34:04.938893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:34:31.651204 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:34:31.651121    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:34:48.643768 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:34:48.643726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:35:10.357876 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:35:10.357839    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:35:36.650757 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:35:36.650721    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:35:56.551685 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:35:56.551655    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:36:08.026157 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:08.026124    2562 scope.go:117] "RemoveContainer" containerID="78d5af8906ea73a6e6ca2d3bcda676775519d37ab0a15d07073b75eeada5f2c9"
Jul 30 04:36:08.034337 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:08.034318    2562 scope.go:117] "RemoveContainer" containerID="ed76dbcfc9a077c90e07c89ebcaae0f10e0e5cd2abd95c0cdeec3bc0875849d4"
Jul 30 04:36:08.041368 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:08.041341    2562 scope.go:117] "RemoveContainer" containerID="d29f31fee941aee228296c135e543ac1eb40682eece58b5e9cbd72a7dc319060"
Jul 30 04:36:23.344292 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:23.344258    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:36:40.450175 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:40.450141    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:36:53.047340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:36:53.047306    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:37:08.068705 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:08.068670    2562 scope.go:117] "RemoveContainer" containerID="a1ff5d05a454ccb89d7b80abd364a90ed54aec0c5d28bc9bb25b6bff61f05465"
Jul 30 04:37:19.945958 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:19.945924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:37:24.893145 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:24.893106    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:37:25.835221 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:25.835186    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:37:36.148008 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:36.147971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:37:39.912501 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:39.912457    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" podUID="6afa3ec8-1447-4059-b275-fd51253aee30" containerName="main" containerID="cri-o://931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b" gracePeriod=45
Jul 30 04:37:40.163489 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.163432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:37:40.188808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.188779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.188942 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.188841    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r257\" (UniqueName: \"kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.188942 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.188885    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.189273 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.189273 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache" (OuterVolumeSpecName: "model-cache") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:40.189273 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.189273 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189246    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs\") pod \"6afa3ec8-1447-4059-b275-fd51253aee30\" (UID: \"6afa3ec8-1447-4059-b275-fd51253aee30\") "
Jul 30 04:37:40.189605 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home" (OuterVolumeSpecName: "home") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:40.189605 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:40.189605 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189567    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.189605 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189585    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.189605 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.189599    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.191069 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.191043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm" (OuterVolumeSpecName: "dshm") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:40.191697 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.191676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:37:40.191782 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.191719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257" (OuterVolumeSpecName: "kube-api-access-5r257") pod "6afa3ec8-1447-4059-b275-fd51253aee30" (UID: "6afa3ec8-1447-4059-b275-fd51253aee30"). InnerVolumeSpecName "kube-api-access-5r257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:37:40.258213 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.258177    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:37:40.258564 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.258550    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa3ec8-1447-4059-b275-fd51253aee30" containerName="main"
Jul 30 04:37:40.258624 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.258566    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa3ec8-1447-4059-b275-fd51253aee30" containerName="main"
Jul 30 04:37:40.258666 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.258629    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa3ec8-1447-4059-b275-fd51253aee30" containerName="main"
Jul 30 04:37:40.261580 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.261564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.263937 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.263916    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"sub-test-model-324a29-kserve-self-signed-certs\""
Jul 30 04:37:40.269905 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.269883    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:37:40.290345 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290479 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ns25\" (UniqueName: \"kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290479 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290479 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290670 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290670 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.290770 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290686    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r257\" (UniqueName: \"kubernetes.io/projected/6afa3ec8-1447-4059-b275-fd51253aee30-kube-api-access-5r257\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.290770 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290706    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afa3ec8-1447-4059-b275-fd51253aee30-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.290770 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.290721    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afa3ec8-1447-4059-b275-fd51253aee30-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:40.391771 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.391904 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.391904 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.391904 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ns25\" (UniqueName: \"kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.392076 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.392076 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.391996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.392347 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.392320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.392434 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.392362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.392725 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.392697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.394303 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.394281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.394446 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.394428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.401861 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.401837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ns25\" (UniqueName: \"kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25\") pod \"sub-test-model-324a29-kserve-77c9b8658f-xwfcv\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") " pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.572041 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.572017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:40.702418 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.702392    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:37:40.704628 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:37:40.704598    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod244da6ea_ac21_4e95_a98b_31576bb84b47.slice/crio-706115f82b26a603e7d4bd92c6141281c1e89ef07aa007f0ce4dde21e1ada620 WatchSource:0}: Error finding container 706115f82b26a603e7d4bd92c6141281c1e89ef07aa007f0ce4dde21e1ada620: Status 404 returned error can't find the container with id 706115f82b26a603e7d4bd92c6141281c1e89ef07aa007f0ce4dde21e1ada620
Jul 30 04:37:40.710378 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.710348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" event={"ID":"244da6ea-ac21-4e95-a98b-31576bb84b47","Type":"ContainerStarted","Data":"706115f82b26a603e7d4bd92c6141281c1e89ef07aa007f0ce4dde21e1ada620"}
Jul 30 04:37:40.711652 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.711627    2562 generic.go:356] "Generic (PLEG): container finished" podID="6afa3ec8-1447-4059-b275-fd51253aee30" containerID="931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b" exitCode=0
Jul 30 04:37:40.711717 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.711679    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"
Jul 30 04:37:40.711774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.711679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" event={"ID":"6afa3ec8-1447-4059-b275-fd51253aee30","Type":"ContainerDied","Data":"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"}
Jul 30 04:37:40.711817 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.711795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj" event={"ID":"6afa3ec8-1447-4059-b275-fd51253aee30","Type":"ContainerDied","Data":"bd9ef1e82e56862e87d0719c739f93356127cb60ee00e2c5903222781599d4b3"}
Jul 30 04:37:40.711857 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.711816    2562 scope.go:117] "RemoveContainer" containerID="931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"
Jul 30 04:37:40.724129 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.724108    2562 scope.go:117] "RemoveContainer" containerID="931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"
Jul 30 04:37:40.724409 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:37:40.724390    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b\": container with ID starting with 931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b not found: ID does not exist" containerID="931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"
Jul 30 04:37:40.724480 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.724421    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b"} err="failed to get container status \"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b\": rpc error: code = NotFound desc = could not find container \"931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b\": container with ID starting with 931021bd1e1bceb63aae7dacee49de6532c3387b8b39b89daf7d4e06a7631e2b not found: ID does not exist"
Jul 30 04:37:40.734260 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.734241    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:37:40.735825 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.735795    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/auth-test-model-324a29-kserve-57b88bd7f5-9vgnj"]
Jul 30 04:37:40.854438 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:40.854359    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" podUID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" containerName="main" containerID="cri-o://c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036" gracePeriod=45
Jul 30 04:37:41.105610 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.105546    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:37:41.198056 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198023    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198102    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c44t7\" (UniqueName: \"kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198177    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198200    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home\") pod \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\" (UID: \"13dadd0b-7c26-4549-83bc-cdea94bc92fe\") "
Jul 30 04:37:41.198489 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:41.198489 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:41.198489 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198476    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.198670 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.198539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home" (OuterVolumeSpecName: "home") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:41.200490 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.200462    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm" (OuterVolumeSpecName: "dshm") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:37:41.200608 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.200568    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7" (OuterVolumeSpecName: "kube-api-access-c44t7") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "kube-api-access-c44t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:37:41.200608 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.200582    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13dadd0b-7c26-4549-83bc-cdea94bc92fe" (UID: "13dadd0b-7c26-4549-83bc-cdea94bc92fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:37:41.299909 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.299876    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c44t7\" (UniqueName: \"kubernetes.io/projected/13dadd0b-7c26-4549-83bc-cdea94bc92fe-kube-api-access-c44t7\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.299909 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.299904    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.299909 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.299913    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.299909 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.299920    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.300163 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.299928    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dadd0b-7c26-4549-83bc-cdea94bc92fe-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:37:41.718176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.718141    2562 generic.go:356] "Generic (PLEG): container finished" podID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" containerID="c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036" exitCode=0
Jul 30 04:37:41.718331 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.718187    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"
Jul 30 04:37:41.718331 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.718225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" event={"ID":"13dadd0b-7c26-4549-83bc-cdea94bc92fe","Type":"ContainerDied","Data":"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"}
Jul 30 04:37:41.718331 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.718265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2" event={"ID":"13dadd0b-7c26-4549-83bc-cdea94bc92fe","Type":"ContainerDied","Data":"1d4ab8c3c634dda4d7cb585f7c921642822d646a2e72448b87fb0d4a6945e498"}
Jul 30 04:37:41.718331 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.718287    2562 scope.go:117] "RemoveContainer" containerID="c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"
Jul 30 04:37:41.720955 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.720928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" event={"ID":"244da6ea-ac21-4e95-a98b-31576bb84b47","Type":"ContainerStarted","Data":"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"}
Jul 30 04:37:41.727650 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.727636    2562 scope.go:117] "RemoveContainer" containerID="c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"
Jul 30 04:37:41.727915 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:37:41.727883    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036\": container with ID starting with c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036 not found: ID does not exist" containerID="c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"
Jul 30 04:37:41.727954 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.727922    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036"} err="failed to get container status \"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036\": rpc error: code = NotFound desc = could not find container \"c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036\": container with ID starting with c80f70071bd96cbb4f4bd003ba4ddb4ca791f6111403f9c7103c587d51511036 not found: ID does not exist"
Jul 30 04:37:41.740316 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.740276    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" podStartSLOduration=1.7402642529999999 podStartE2EDuration="1.740264253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:37:41.738926886 +0000 UTC m=+3154.555863363" watchObservedRunningTime="2026-07-30 04:37:41.740264253 +0000 UTC m=+3154.557200748"
Jul 30 04:37:41.752170 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.752142    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:37:41.753792 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.753774    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/auth-test-model-64af9f-kserve-59694ff9f5-ktbs2"]
Jul 30 04:37:41.785346 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.785326    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" path="/var/lib/kubelet/pods/13dadd0b-7c26-4549-83bc-cdea94bc92fe/volumes"
Jul 30 04:37:41.785723 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:41.785710    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afa3ec8-1447-4059-b275-fd51253aee30" path="/var/lib/kubelet/pods/6afa3ec8-1447-4059-b275-fd51253aee30/volumes"
Jul 30 04:37:45.646940 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:45.646906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:37:50.572628 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:50.572593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:50.573061 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:50.572648    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:50.585357 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:50.585329    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:37:50.775479 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:37:50.775452    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:38:33.746727 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.746696    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:38:33.747188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.747041    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" containerName="main"
Jul 30 04:38:33.747188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.747053    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" containerName="main"
Jul 30 04:38:33.747188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.747119    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13dadd0b-7c26-4549-83bc-cdea94bc92fe" containerName="main"
Jul 30 04:38:33.749984 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.749968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.753407 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.753386    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"openshift-service-ca.crt\""
Jul 30 04:38:33.753565 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.753411    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"default-dockercfg-d52rx\""
Jul 30 04:38:33.753565 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.753435    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"sub-test-model-64af9f-kserve-self-signed-certs\""
Jul 30 04:38:33.753665 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.753577    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"kube-root-ca.crt\""
Jul 30 04:38:33.758211 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.758191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:38:33.901458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.901458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.901729 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7fb\" (UniqueName: \"kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.901729 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.901729 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:33.901853 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:33.901748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.002745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7fb\" (UniqueName: \"kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.002745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.002745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003040 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003040 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003040 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.002941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003294 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.003262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003416 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.003300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.003416 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.003323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.004901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.004882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.005013 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.004997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.009919 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.009898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7fb\" (UniqueName: \"kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb\") pod \"sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") " pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.060912 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.060889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:34.189337 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.189310    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:38:34.192695 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:38:34.192661    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e9bd88_36b7_456c_a414_b60763961815.slice/crio-1e5166010c17f766471fe97f9fc7c914a2652ff4ca28e9c89d0a6f9fb54b36a3 WatchSource:0}: Error finding container 1e5166010c17f766471fe97f9fc7c914a2652ff4ca28e9c89d0a6f9fb54b36a3: Status 404 returned error can't find the container with id 1e5166010c17f766471fe97f9fc7c914a2652ff4ca28e9c89d0a6f9fb54b36a3
Jul 30 04:38:34.966593 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.966558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" event={"ID":"12e9bd88-36b7-456c-a414-b60763961815","Type":"ContainerStarted","Data":"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"}
Jul 30 04:38:34.966593 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.966599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" event={"ID":"12e9bd88-36b7-456c-a414-b60763961815","Type":"ContainerStarted","Data":"1e5166010c17f766471fe97f9fc7c914a2652ff4ca28e9c89d0a6f9fb54b36a3"}
Jul 30 04:38:34.985147 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:34.985102    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" podStartSLOduration=1.9850877759999999 podStartE2EDuration="1.985087776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:38:34.984329588 +0000 UTC m=+3207.801266066" watchObservedRunningTime="2026-07-30 04:38:34.985087776 +0000 UTC m=+3207.802024276"
Jul 30 04:38:37.244850 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:37.244816    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:38:44.061635 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:44.061593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:44.061635 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:44.061644    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:44.074906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:44.074872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:38:45.025173 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:38:45.025140    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:39:25.748622 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:39:25.748585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:39:42.538549 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:39:42.538503    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:40:06.551982 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:06.551951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:40:12.166729 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:12.166692    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:40:13.053901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:13.053863    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:40:22.840145 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.840112    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:40:22.843894 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.843872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.846152 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.846133    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-324a29\"/\"rate-test-model-324a29-kserve-self-signed-certs\""
Jul 30 04:40:22.852969 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.852947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:40:22.962133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.962293 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.962293 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.962293 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4762\" (UniqueName: \"kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.962293 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:22.962476 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:22.962337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063440 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4762\" (UniqueName: \"kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.063917 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.064016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.064016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.063989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.065906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.065888    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.066174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.066153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.071931 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.071904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4762\" (UniqueName: \"kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762\") pod \"rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") " pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.154661 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.154608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:23.286106 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.286080    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:40:23.287899 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:40:23.287861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c705d7d_a38c_4f61_8232_fa6acd8db571.slice/crio-14ea0eb666093d44699223484de087ec0e1e743e9660d6f34a0020fb42aa74ee WatchSource:0}: Error finding container 14ea0eb666093d44699223484de087ec0e1e743e9660d6f34a0020fb42aa74ee: Status 404 returned error can't find the container with id 14ea0eb666093d44699223484de087ec0e1e743e9660d6f34a0020fb42aa74ee
Jul 30 04:40:23.476831 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.476796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" event={"ID":"5c705d7d-a38c-4f61-8232-fa6acd8db571","Type":"ContainerStarted","Data":"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"}
Jul 30 04:40:23.476990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.476839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" event={"ID":"5c705d7d-a38c-4f61-8232-fa6acd8db571","Type":"ContainerStarted","Data":"14ea0eb666093d44699223484de087ec0e1e743e9660d6f34a0020fb42aa74ee"}
Jul 30 04:40:23.492959 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:23.492903    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" podStartSLOduration=1.492883411 podStartE2EDuration="1.492883411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:40:23.491740442 +0000 UTC m=+3316.308676921" watchObservedRunningTime="2026-07-30 04:40:23.492883411 +0000 UTC m=+3316.309819889"
Jul 30 04:40:26.442618 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:26.442582    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:40:27.186106 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.186065    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" podUID="244da6ea-ac21-4e95-a98b-31576bb84b47" containerName="main" containerID="cri-o://55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f" gracePeriod=45
Jul 30 04:40:27.433000 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.432977    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:40:27.498787 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.498762    2562 generic.go:356] "Generic (PLEG): container finished" podID="244da6ea-ac21-4e95-a98b-31576bb84b47" containerID="55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f" exitCode=0
Jul 30 04:40:27.499174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.498811    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"
Jul 30 04:40:27.499174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.498824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" event={"ID":"244da6ea-ac21-4e95-a98b-31576bb84b47","Type":"ContainerDied","Data":"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"}
Jul 30 04:40:27.499174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.498854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv" event={"ID":"244da6ea-ac21-4e95-a98b-31576bb84b47","Type":"ContainerDied","Data":"706115f82b26a603e7d4bd92c6141281c1e89ef07aa007f0ce4dde21e1ada620"}
Jul 30 04:40:27.499174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.498875    2562 scope.go:117] "RemoveContainer" containerID="55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"
Jul 30 04:40:27.501897 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.501879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502010 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.501931    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502010 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.501980    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ns25\" (UniqueName: \"kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502123 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502123 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502225 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:27.502278 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache" (OuterVolumeSpecName: "model-cache") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:27.502379 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home" (OuterVolumeSpecName: "home") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:27.502379 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502364    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs\") pod \"244da6ea-ac21-4e95-a98b-31576bb84b47\" (UID: \"244da6ea-ac21-4e95-a98b-31576bb84b47\") "
Jul 30 04:40:27.502638 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502622    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.502718 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502644    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.502718 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.502656    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.504461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.504437    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:40:27.504636 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.504609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm" (OuterVolumeSpecName: "dshm") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:27.504689 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.504638    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25" (OuterVolumeSpecName: "kube-api-access-6ns25") pod "244da6ea-ac21-4e95-a98b-31576bb84b47" (UID: "244da6ea-ac21-4e95-a98b-31576bb84b47"). InnerVolumeSpecName "kube-api-access-6ns25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:27.507504 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.507488    2562 scope.go:117] "RemoveContainer" containerID="55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"
Jul 30 04:40:27.507813 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:40:27.507785    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f\": container with ID starting with 55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f not found: ID does not exist" containerID="55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"
Jul 30 04:40:27.507874 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.507830    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f"} err="failed to get container status \"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f\": rpc error: code = NotFound desc = could not find container \"55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f\": container with ID starting with 55b9a507d1b820c6ac0b72af7f4ef0136b0a5d714f9a4115b5b0811c17d32d2f not found: ID does not exist"
Jul 30 04:40:27.603396 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.603374    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ns25\" (UniqueName: \"kubernetes.io/projected/244da6ea-ac21-4e95-a98b-31576bb84b47-kube-api-access-6ns25\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.603396 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.603396    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244da6ea-ac21-4e95-a98b-31576bb84b47-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.603541 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.603405    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244da6ea-ac21-4e95-a98b-31576bb84b47-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:27.813586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.813556    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:40:27.816859 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:27.816838    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/sub-test-model-324a29-kserve-77c9b8658f-xwfcv"]
Jul 30 04:40:28.073126 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.073063    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" podUID="12e9bd88-36b7-456c-a414-b60763961815" containerName="main" containerID="cri-o://7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41" gracePeriod=45
Jul 30 04:40:28.329451 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.329386    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:40:28.408906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.408880    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409024 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.408916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409024 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.408935    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm7fb\" (UniqueName: \"kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409024 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.408964    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409024 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.408987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409247 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409043    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm\") pod \"12e9bd88-36b7-456c-a414-b60763961815\" (UID: \"12e9bd88-36b7-456c-a414-b60763961815\") "
Jul 30 04:40:28.409247 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409149    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache" (OuterVolumeSpecName: "model-cache") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:28.409247 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409182    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home" (OuterVolumeSpecName: "home") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:28.409247 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:28.409429 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409321    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.409429 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409334    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.409429 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.409343    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.410993 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.410965    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm" (OuterVolumeSpecName: "dshm") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:40:28.411090 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.411013    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:40:28.411090 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.411062    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb" (OuterVolumeSpecName: "kube-api-access-wm7fb") pod "12e9bd88-36b7-456c-a414-b60763961815" (UID: "12e9bd88-36b7-456c-a414-b60763961815"). InnerVolumeSpecName "kube-api-access-wm7fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:40:28.509877 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.509846    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7fb\" (UniqueName: \"kubernetes.io/projected/12e9bd88-36b7-456c-a414-b60763961815-kube-api-access-wm7fb\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.509877 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.509872    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e9bd88-36b7-456c-a414-b60763961815-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.510305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.509888    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e9bd88-36b7-456c-a414-b60763961815-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:40:28.512458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.512426    2562 generic.go:356] "Generic (PLEG): container finished" podID="12e9bd88-36b7-456c-a414-b60763961815" containerID="7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41" exitCode=0
Jul 30 04:40:28.512597 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.512488    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"
Jul 30 04:40:28.512597 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.512540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" event={"ID":"12e9bd88-36b7-456c-a414-b60763961815","Type":"ContainerDied","Data":"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"}
Jul 30 04:40:28.512597 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.512582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4" event={"ID":"12e9bd88-36b7-456c-a414-b60763961815","Type":"ContainerDied","Data":"1e5166010c17f766471fe97f9fc7c914a2652ff4ca28e9c89d0a6f9fb54b36a3"}
Jul 30 04:40:28.512828 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.512603    2562 scope.go:117] "RemoveContainer" containerID="7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"
Jul 30 04:40:28.522417 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.522402    2562 scope.go:117] "RemoveContainer" containerID="7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"
Jul 30 04:40:28.522722 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:40:28.522701    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41\": container with ID starting with 7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41 not found: ID does not exist" containerID="7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"
Jul 30 04:40:28.522779 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.522729    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41"} err="failed to get container status \"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41\": rpc error: code = NotFound desc = could not find container \"7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41\": container with ID starting with 7ea04914daa18f727defd90788699e95b7883faa3ef8d92b746afab422f39f41 not found: ID does not exist"
Jul 30 04:40:28.533792 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.533771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:40:28.539754 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:28.539734    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/sub-test-model-64af9f-kserve-9cbd7df6c-qcbm4"]
Jul 30 04:40:29.785465 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:29.785386    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e9bd88-36b7-456c-a414-b60763961815" path="/var/lib/kubelet/pods/12e9bd88-36b7-456c-a414-b60763961815/volumes"
Jul 30 04:40:29.785829 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:29.785781    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244da6ea-ac21-4e95-a98b-31576bb84b47" path="/var/lib/kubelet/pods/244da6ea-ac21-4e95-a98b-31576bb84b47/volumes"
Jul 30 04:40:33.155129 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:33.155095    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:33.155551 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:33.155166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:33.167901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:33.167880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:33.547701 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:33.547675    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:40:58.248922 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.248891    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249250    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e9bd88-36b7-456c-a414-b60763961815" containerName="main"
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e9bd88-36b7-456c-a414-b60763961815" containerName="main"
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249277    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244da6ea-ac21-4e95-a98b-31576bb84b47" containerName="main"
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249283    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="244da6ea-ac21-4e95-a98b-31576bb84b47" containerName="main"
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249355    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="244da6ea-ac21-4e95-a98b-31576bb84b47" containerName="main"
Jul 30 04:40:58.249384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.249366    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e9bd88-36b7-456c-a414-b60763961815" containerName="main"
Jul 30 04:40:58.252417 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.252402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.255067 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.255007    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"openshift-service-ca.crt\""
Jul 30 04:40:58.256067 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.256046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"default-dockercfg-d52rx\""
Jul 30 04:40:58.256195 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.256047    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"kube-root-ca.crt\""
Jul 30 04:40:58.256195 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.256154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-64af9f\"/\"rate-test-model-64af9f-kserve-self-signed-certs\""
Jul 30 04:40:58.263139 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.263097    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:40:58.325461 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.325598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnww7\" (UniqueName: \"kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.325598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.325598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.325598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.325758 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.325606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426540 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426661 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426661 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnww7\" (UniqueName: \"kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426661 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426831 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.426831 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.427021 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.427120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.426996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.427120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.427102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.428855 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.428834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.429221 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.429200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.433970 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.433952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnww7\" (UniqueName: \"kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7\") pod \"rate-test-model-64af9f-kserve-b944ffb59-56qnz\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") " pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.563033 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.562974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:40:58.693168 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:58.693143    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:40:58.694799 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:40:58.694770    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c00c8ce_a942_49a7_bd0f_9a414910819e.slice/crio-9dca9a27bcff5bf8a4971a1254a17d2adc860752e9eb6f9bc60b24a8e089eceb WatchSource:0}: Error finding container 9dca9a27bcff5bf8a4971a1254a17d2adc860752e9eb6f9bc60b24a8e089eceb: Status 404 returned error can't find the container with id 9dca9a27bcff5bf8a4971a1254a17d2adc860752e9eb6f9bc60b24a8e089eceb
Jul 30 04:40:59.668622 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:59.668582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" event={"ID":"0c00c8ce-a942-49a7-bd0f-9a414910819e","Type":"ContainerStarted","Data":"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"}
Jul 30 04:40:59.669012 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:59.668630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" event={"ID":"0c00c8ce-a942-49a7-bd0f-9a414910819e","Type":"ContainerStarted","Data":"9dca9a27bcff5bf8a4971a1254a17d2adc860752e9eb6f9bc60b24a8e089eceb"}
Jul 30 04:40:59.686535 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:40:59.686463    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" podStartSLOduration=1.686448261 podStartE2EDuration="1.686448261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:40:59.684188794 +0000 UTC m=+3352.501125284" watchObservedRunningTime="2026-07-30 04:40:59.686448261 +0000 UTC m=+3352.503384738"
Jul 30 04:41:01.642173 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:01.642139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:41:08.563072 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:08.563040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:41:08.563432 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:08.563093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:41:08.575302 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:08.575278    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:41:08.724665 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:08.724638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:41:52.444538 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:41:52.444485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:42:08.755233 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:08.755189    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:42:44.146580 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:44.146537    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:42:50.624026 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:50.623993    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:42:51.531371 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:51.531339    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:42:52.491306 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.491273    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140636230649776"]
Jul 30 04:42:52.494280 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.494263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:52.496623 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.496599    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dzc85\""
Jul 30 04:42:52.500011 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.499990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230649776"]
Jul 30 04:42:52.588115 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.588090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfbqw\" (UniqueName: \"kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw\") pod \"test-curl-7753-140636230649776\" (UID: \"06278c8c-4921-4eb9-98d7-7ce72a27f7d7\") " pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:52.688996 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.688962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfbqw\" (UniqueName: \"kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw\") pod \"test-curl-7753-140636230649776\" (UID: \"06278c8c-4921-4eb9-98d7-7ce72a27f7d7\") " pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:52.697102 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.697078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfbqw\" (UniqueName: \"kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw\") pod \"test-curl-7753-140636230649776\" (UID: \"06278c8c-4921-4eb9-98d7-7ce72a27f7d7\") " pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:52.805384 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.805334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:52.930096 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.930066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230649776"]
Jul 30 04:42:52.932454 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:42:52.932429    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06278c8c_4921_4eb9_98d7_7ce72a27f7d7.slice/crio-4ded9fdf1ebef73280a6806309afe25cefe6ae20564c9b1e1ece0eca2cfbc6b0 WatchSource:0}: Error finding container 4ded9fdf1ebef73280a6806309afe25cefe6ae20564c9b1e1ece0eca2cfbc6b0: Status 404 returned error can't find the container with id 4ded9fdf1ebef73280a6806309afe25cefe6ae20564c9b1e1ece0eca2cfbc6b0
Jul 30 04:42:52.933892 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:52.933875    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:42:53.210913 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:53.210830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230649776" event={"ID":"06278c8c-4921-4eb9-98d7-7ce72a27f7d7","Type":"ContainerStarted","Data":"4ded9fdf1ebef73280a6806309afe25cefe6ae20564c9b1e1ece0eca2cfbc6b0"}
Jul 30 04:42:54.217275 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:54.217231    2562 generic.go:356] "Generic (PLEG): container finished" podID="06278c8c-4921-4eb9-98d7-7ce72a27f7d7" containerID="6ae068d05622cb68e2a372453163a1754a9a0c646f966f1650460f3ee4fb663a" exitCode=0
Jul 30 04:42:54.217275 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:54.217256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230649776" event={"ID":"06278c8c-4921-4eb9-98d7-7ce72a27f7d7","Type":"ContainerDied","Data":"6ae068d05622cb68e2a372453163a1754a9a0c646f966f1650460f3ee4fb663a"}
Jul 30 04:42:55.351817 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.351792    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:42:55.388781 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.388753    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140636230649776_06278c8c-4921-4eb9-98d7-7ce72a27f7d7/test-curl-7753-140636230649776/0.log"
Jul 30 04:42:55.413431 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.413397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140636230649776"]
Jul 30 04:42:55.417041 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.417014    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140636230649776"]
Jul 30 04:42:55.512758 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.512725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfbqw\" (UniqueName: \"kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw\") pod \"06278c8c-4921-4eb9-98d7-7ce72a27f7d7\" (UID: \"06278c8c-4921-4eb9-98d7-7ce72a27f7d7\") "
Jul 30 04:42:55.514901 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.514866    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw" (OuterVolumeSpecName: "kube-api-access-xfbqw") pod "06278c8c-4921-4eb9-98d7-7ce72a27f7d7" (UID: "06278c8c-4921-4eb9-98d7-7ce72a27f7d7"). InnerVolumeSpecName "kube-api-access-xfbqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:42:55.614190 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.614165    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfbqw\" (UniqueName: \"kubernetes.io/projected/06278c8c-4921-4eb9-98d7-7ce72a27f7d7-kube-api-access-xfbqw\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:42:55.785621 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:55.785552    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06278c8c-4921-4eb9-98d7-7ce72a27f7d7" path="/var/lib/kubelet/pods/06278c8c-4921-4eb9-98d7-7ce72a27f7d7/volumes"
Jul 30 04:42:56.228007 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:56.227976    2562 scope.go:117] "RemoveContainer" containerID="6ae068d05622cb68e2a372453163a1754a9a0c646f966f1650460f3ee4fb663a"
Jul 30 04:42:56.228173 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:42:56.227981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230649776"
Jul 30 04:43:02.013273 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.013240    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140636230491280"]
Jul 30 04:43:02.013660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.013587    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06278c8c-4921-4eb9-98d7-7ce72a27f7d7" containerName="test-curl-7753-140636230649776"
Jul 30 04:43:02.013660 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.013600    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="06278c8c-4921-4eb9-98d7-7ce72a27f7d7" containerName="test-curl-7753-140636230649776"
Jul 30 04:43:02.013737 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.013675    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="06278c8c-4921-4eb9-98d7-7ce72a27f7d7" containerName="test-curl-7753-140636230649776"
Jul 30 04:43:02.017920 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.017900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:02.019878 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.019852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230491280"]
Jul 30 04:43:02.020328 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.020310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dzc85\""
Jul 30 04:43:02.062280 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.062251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mx2\" (UniqueName: \"kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2\") pod \"test-curl-7753-140636230491280\" (UID: \"9f0fea8f-ea67-4b1c-90ac-c5197586423c\") " pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:02.144448 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.144419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:43:02.162875 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.162840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mx2\" (UniqueName: \"kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2\") pod \"test-curl-7753-140636230491280\" (UID: \"9f0fea8f-ea67-4b1c-90ac-c5197586423c\") " pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:02.170462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.170432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mx2\" (UniqueName: \"kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2\") pod \"test-curl-7753-140636230491280\" (UID: \"9f0fea8f-ea67-4b1c-90ac-c5197586423c\") " pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:02.327661 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.327592    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:02.455466 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:02.455443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230491280"]
Jul 30 04:43:02.458877 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:43:02.458849    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f0fea8f_ea67_4b1c_90ac_c5197586423c.slice/crio-37a1245acfad578a8f349a91d0fcc100c1d5d4dfe26806b96597f5727973f3be WatchSource:0}: Error finding container 37a1245acfad578a8f349a91d0fcc100c1d5d4dfe26806b96597f5727973f3be: Status 404 returned error can't find the container with id 37a1245acfad578a8f349a91d0fcc100c1d5d4dfe26806b96597f5727973f3be
Jul 30 04:43:03.263125 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:03.263092    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" containerID="582490cb39755084b6403112a8e9b1104ff87f14b16a4a1f7a172c574ad326f8" exitCode=0
Jul 30 04:43:03.263481 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:03.263149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230491280" event={"ID":"9f0fea8f-ea67-4b1c-90ac-c5197586423c","Type":"ContainerDied","Data":"582490cb39755084b6403112a8e9b1104ff87f14b16a4a1f7a172c574ad326f8"}
Jul 30 04:43:03.263481 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:03.263177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230491280" event={"ID":"9f0fea8f-ea67-4b1c-90ac-c5197586423c","Type":"ContainerStarted","Data":"37a1245acfad578a8f349a91d0fcc100c1d5d4dfe26806b96597f5727973f3be"}
Jul 30 04:43:04.402954 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.402930    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:04.421601 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.421576    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140636230491280_9f0fea8f-ea67-4b1c-90ac-c5197586423c/test-curl-7753-140636230491280/0.log"
Jul 30 04:43:04.447274 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.447248    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140636230491280"]
Jul 30 04:43:04.449792 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.449766    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140636230491280"]
Jul 30 04:43:04.481309 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.481288    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4mx2\" (UniqueName: \"kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2\") pod \"9f0fea8f-ea67-4b1c-90ac-c5197586423c\" (UID: \"9f0fea8f-ea67-4b1c-90ac-c5197586423c\") "
Jul 30 04:43:04.483309 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.483280    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2" (OuterVolumeSpecName: "kube-api-access-m4mx2") pod "9f0fea8f-ea67-4b1c-90ac-c5197586423c" (UID: "9f0fea8f-ea67-4b1c-90ac-c5197586423c"). InnerVolumeSpecName "kube-api-access-m4mx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:43:04.582636 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:04.582586    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mx2\" (UniqueName: \"kubernetes.io/projected/9f0fea8f-ea67-4b1c-90ac-c5197586423c-kube-api-access-m4mx2\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:05.274983 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.274955    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:05.274983 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.274966    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a1245acfad578a8f349a91d0fcc100c1d5d4dfe26806b96597f5727973f3be"
Jul 30 04:43:05.284981 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:05.284955    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7753-140636230491280\" is forbidden: User \"system:node:ip-10-0-138-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-44.ec2.internal' and this object" podUID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" pod="opendatahub/test-curl-7753-140636230491280"
Jul 30 04:43:05.642207 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.642119    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" podUID="5c705d7d-a38c-4f61-8232-fa6acd8db571" containerName="main" containerID="cri-o://acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716" gracePeriod=45
Jul 30 04:43:05.785922 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.785887    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" path="/var/lib/kubelet/pods/9f0fea8f-ea67-4b1c-90ac-c5197586423c/volumes"
Jul 30 04:43:05.876152 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.876129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:43:05.995381 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995546 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995390    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995546 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4762\" (UniqueName: \"kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995546 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995439    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995546 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995476    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995546 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir\") pod \"5c705d7d-a38c-4f61-8232-fa6acd8db571\" (UID: \"5c705d7d-a38c-4f61-8232-fa6acd8db571\") "
Jul 30 04:43:05.995826 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995637    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:05.995826 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home" (OuterVolumeSpecName: "home") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:05.995929 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995893    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:05.995929 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995910    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:05.996021 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.995942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:05.997573 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.997543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:43:05.997843 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.997823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm" (OuterVolumeSpecName: "dshm") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:05.997922 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:05.997863    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762" (OuterVolumeSpecName: "kube-api-access-p4762") pod "5c705d7d-a38c-4f61-8232-fa6acd8db571" (UID: "5c705d7d-a38c-4f61-8232-fa6acd8db571"). InnerVolumeSpecName "kube-api-access-p4762". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:43:06.096272 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.096247    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c705d7d-a38c-4f61-8232-fa6acd8db571-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.096369 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.096290    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.096369 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.096299    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c705d7d-a38c-4f61-8232-fa6acd8db571-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.096369 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.096308    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4762\" (UniqueName: \"kubernetes.io/projected/5c705d7d-a38c-4f61-8232-fa6acd8db571-kube-api-access-p4762\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.281011 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.280926    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c705d7d-a38c-4f61-8232-fa6acd8db571" containerID="acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716" exitCode=0
Jul 30 04:43:06.281011 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.280967    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"
Jul 30 04:43:06.281188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.281013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" event={"ID":"5c705d7d-a38c-4f61-8232-fa6acd8db571","Type":"ContainerDied","Data":"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"}
Jul 30 04:43:06.281188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.281061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l" event={"ID":"5c705d7d-a38c-4f61-8232-fa6acd8db571","Type":"ContainerDied","Data":"14ea0eb666093d44699223484de087ec0e1e743e9660d6f34a0020fb42aa74ee"}
Jul 30 04:43:06.281188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.281083    2562 scope.go:117] "RemoveContainer" containerID="acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"
Jul 30 04:43:06.290032 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.290014    2562 scope.go:117] "RemoveContainer" containerID="acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"
Jul 30 04:43:06.290306 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:06.290284    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716\": container with ID starting with acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716 not found: ID does not exist" containerID="acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"
Jul 30 04:43:06.290361 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.290318    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716"} err="failed to get container status \"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716\": rpc error: code = NotFound desc = could not find container \"acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716\": container with ID starting with acd5e09f2e97f17b140176189b0f6ac8dc9f3fdc26bf1ac5c56c8be6db779716 not found: ID does not exist"
Jul 30 04:43:06.304071 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.304047    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:43:06.310352 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.310328    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-324a29/rate-test-model-324a29-kserve-84cdc5c8d7-wcg5l"]
Jul 30 04:43:06.550474 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.550379    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" podUID="0c00c8ce-a942-49a7-bd0f-9a414910819e" containerName="main" containerID="cri-o://ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3" gracePeriod=45
Jul 30 04:43:06.798009 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.797983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:43:06.902889 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.902804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.902889 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.902844    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.902889 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.902868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnww7\" (UniqueName: \"kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.903190 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.902909    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.903190 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.902960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.903190 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir\") pod \"0c00c8ce-a942-49a7-bd0f-9a414910819e\" (UID: \"0c00c8ce-a942-49a7-bd0f-9a414910819e\") "
Jul 30 04:43:06.903190 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home" (OuterVolumeSpecName: "home") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:06.903437 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:06.903437 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:06.903611 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903591    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.903611 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903612    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.903789 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.903625    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:06.905016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.904991    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:43:06.905392 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.905371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7" (OuterVolumeSpecName: "kube-api-access-rnww7") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "kube-api-access-rnww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:43:06.905585 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:06.905565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm" (OuterVolumeSpecName: "dshm") pod "0c00c8ce-a942-49a7-bd0f-9a414910819e" (UID: "0c00c8ce-a942-49a7-bd0f-9a414910819e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:43:07.005016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.004988    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00c8ce-a942-49a7-bd0f-9a414910819e-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:07.005016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.005013    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnww7\" (UniqueName: \"kubernetes.io/projected/0c00c8ce-a942-49a7-bd0f-9a414910819e-kube-api-access-rnww7\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:07.005016 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.005022    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00c8ce-a942-49a7-bd0f-9a414910819e-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:07.036976 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.036946    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140636230459440"]
Jul 30 04:43:07.037367 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037349    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c00c8ce-a942-49a7-bd0f-9a414910819e" containerName="main"
Jul 30 04:43:07.037409 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037372    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c00c8ce-a942-49a7-bd0f-9a414910819e" containerName="main"
Jul 30 04:43:07.037409 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037392    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c705d7d-a38c-4f61-8232-fa6acd8db571" containerName="main"
Jul 30 04:43:07.037409 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037400    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c705d7d-a38c-4f61-8232-fa6acd8db571" containerName="main"
Jul 30 04:43:07.037507 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037409    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" containerName="test-curl-7753-140636230491280"
Jul 30 04:43:07.037507 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037417    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" containerName="test-curl-7753-140636230491280"
Jul 30 04:43:07.037507 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037492    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c00c8ce-a942-49a7-bd0f-9a414910819e" containerName="main"
Jul 30 04:43:07.037507 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037500    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c705d7d-a38c-4f61-8232-fa6acd8db571" containerName="main"
Jul 30 04:43:07.037507 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.037506    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0fea8f-ea67-4b1c-90ac-c5197586423c" containerName="test-curl-7753-140636230491280"
Jul 30 04:43:07.040510 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.040495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:07.042758 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.042740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dzc85\""
Jul 30 04:43:07.045886 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.045862    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230459440"]
Jul 30 04:43:07.105614 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.105586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxdp\" (UniqueName: \"kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp\") pod \"test-curl-7753-140636230459440\" (UID: \"95d55f48-9764-488c-8dec-eeacf856c2b9\") " pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:07.206152 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.206068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxdp\" (UniqueName: \"kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp\") pod \"test-curl-7753-140636230459440\" (UID: \"95d55f48-9764-488c-8dec-eeacf856c2b9\") " pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:07.213388 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.213365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxdp\" (UniqueName: \"kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp\") pod \"test-curl-7753-140636230459440\" (UID: \"95d55f48-9764-488c-8dec-eeacf856c2b9\") " pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:07.288052 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.288023    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c00c8ce-a942-49a7-bd0f-9a414910819e" containerID="ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3" exitCode=0
Jul 30 04:43:07.288184 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.288076    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"
Jul 30 04:43:07.288184 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.288109    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" event={"ID":"0c00c8ce-a942-49a7-bd0f-9a414910819e","Type":"ContainerDied","Data":"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"}
Jul 30 04:43:07.288184 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.288159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz" event={"ID":"0c00c8ce-a942-49a7-bd0f-9a414910819e","Type":"ContainerDied","Data":"9dca9a27bcff5bf8a4971a1254a17d2adc860752e9eb6f9bc60b24a8e089eceb"}
Jul 30 04:43:07.288184 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.288182    2562 scope.go:117] "RemoveContainer" containerID="ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"
Jul 30 04:43:07.296663 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.296644    2562 scope.go:117] "RemoveContainer" containerID="ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"
Jul 30 04:43:07.296991 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:07.296971    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3\": container with ID starting with ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3 not found: ID does not exist" containerID="ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"
Jul 30 04:43:07.297085 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.297001    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3"} err="failed to get container status \"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3\": rpc error: code = NotFound desc = could not find container \"ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3\": container with ID starting with ef8731889d88f69db61c3a3e5e5409ef98e09efa2cb68d7d073366753565b0f3 not found: ID does not exist"
Jul 30 04:43:07.309967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.309942    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:43:07.312949 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.312930    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-64af9f/rate-test-model-64af9f-kserve-b944ffb59-56qnz"]
Jul 30 04:43:07.350980 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.350962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:07.475954 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.475928    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140636230459440"]
Jul 30 04:43:07.479775 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:43:07.479750    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d55f48_9764_488c_8dec_eeacf856c2b9.slice/crio-d80fac430da53b4be526fbac7e0fc1077a1da3e78c10790c0a26066f7ac7c8dd WatchSource:0}: Error finding container d80fac430da53b4be526fbac7e0fc1077a1da3e78c10790c0a26066f7ac7c8dd: Status 404 returned error can't find the container with id d80fac430da53b4be526fbac7e0fc1077a1da3e78c10790c0a26066f7ac7c8dd
Jul 30 04:43:07.785483 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.785458    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c00c8ce-a942-49a7-bd0f-9a414910819e" path="/var/lib/kubelet/pods/0c00c8ce-a942-49a7-bd0f-9a414910819e/volumes"
Jul 30 04:43:07.785887 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:07.785872    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c705d7d-a38c-4f61-8232-fa6acd8db571" path="/var/lib/kubelet/pods/5c705d7d-a38c-4f61-8232-fa6acd8db571/volumes"
Jul 30 04:43:07.887823 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:07.887796    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95d55f48_9764_488c_8dec_eeacf856c2b9.slice/crio-conmon-e39c25748613290bfaef2046474a49de8849b5523c38724d18465d9a01de3adb.scope\": RecentStats: unable to find data in memory cache]"
Jul 30 04:43:08.293462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:08.293436    2562 generic.go:356] "Generic (PLEG): container finished" podID="95d55f48-9764-488c-8dec-eeacf856c2b9" containerID="e39c25748613290bfaef2046474a49de8849b5523c38724d18465d9a01de3adb" exitCode=0
Jul 30 04:43:08.293572 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:08.293481    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230459440" event={"ID":"95d55f48-9764-488c-8dec-eeacf856c2b9","Type":"ContainerDied","Data":"e39c25748613290bfaef2046474a49de8849b5523c38724d18465d9a01de3adb"}
Jul 30 04:43:08.293572 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:08.293505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140636230459440" event={"ID":"95d55f48-9764-488c-8dec-eeacf856c2b9","Type":"ContainerStarted","Data":"d80fac430da53b4be526fbac7e0fc1077a1da3e78c10790c0a26066f7ac7c8dd"}
Jul 30 04:43:09.432312 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.432288    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:09.451932 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.451906    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140636230459440_95d55f48-9764-488c-8dec-eeacf856c2b9/test-curl-7753-140636230459440/0.log"
Jul 30 04:43:09.477274 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.477248    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140636230459440"]
Jul 30 04:43:09.479971 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.479952    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140636230459440"]
Jul 30 04:43:09.524402 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.524378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mxdp\" (UniqueName: \"kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp\") pod \"95d55f48-9764-488c-8dec-eeacf856c2b9\" (UID: \"95d55f48-9764-488c-8dec-eeacf856c2b9\") "
Jul 30 04:43:09.526370 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.526346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp" (OuterVolumeSpecName: "kube-api-access-5mxdp") pod "95d55f48-9764-488c-8dec-eeacf856c2b9" (UID: "95d55f48-9764-488c-8dec-eeacf856c2b9"). InnerVolumeSpecName "kube-api-access-5mxdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:43:09.624870 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.624808    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxdp\" (UniqueName: \"kubernetes.io/projected/95d55f48-9764-488c-8dec-eeacf856c2b9-kube-api-access-5mxdp\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:43:09.785036 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:09.784997    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d55f48-9764-488c-8dec-eeacf856c2b9" path="/var/lib/kubelet/pods/95d55f48-9764-488c-8dec-eeacf856c2b9/volumes"
Jul 30 04:43:10.303958 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:10.303919    2562 scope.go:117] "RemoveContainer" containerID="e39c25748613290bfaef2046474a49de8849b5523c38724d18465d9a01de3adb"
Jul 30 04:43:10.303958 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:10.303940    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140636230459440"
Jul 30 04:43:14.774179 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.774141    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:43:14.774703 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.774684    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d55f48-9764-488c-8dec-eeacf856c2b9" containerName="test-curl-7753-140636230459440"
Jul 30 04:43:14.774759 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.774707    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d55f48-9764-488c-8dec-eeacf856c2b9" containerName="test-curl-7753-140636230459440"
Jul 30 04:43:14.774826 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.774814    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d55f48-9764-488c-8dec-eeacf856c2b9" containerName="test-curl-7753-140636230459440"
Jul 30 04:43:14.779494 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.779467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.782196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.782175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a1808a-serving-cert\""
Jul 30 04:43:14.782196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.782190    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 30 04:43:14.782196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.782202    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 04:43:14.782440 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.782232    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lzvw2\""
Jul 30 04:43:14.787733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.787537    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:43:14.864588 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.864562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4v4\" (UniqueName: \"kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.864758 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.864726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.864883 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.864865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.965365 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.965342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4v4\" (UniqueName: \"kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.965618 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.965377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.965618 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.965415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.966076 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.966057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.968054 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.968031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:14.973459 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:14.973439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4v4\" (UniqueName: \"kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4\") pod \"maas-api-e2e-ipp-a-a1808a-68df456968-gbt42\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:15.092139 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:15.092062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:15.228042 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:15.228016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:43:15.229899 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:43:15.229869    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9407b4b_76fd_47a1_8f15_718ca5d4f993.slice/crio-fcc9b182716c4dcd335ec7d42a6c73c11092c336c68cd6b7f288290808d2e406 WatchSource:0}: Error finding container fcc9b182716c4dcd335ec7d42a6c73c11092c336c68cd6b7f288290808d2e406: Status 404 returned error can't find the container with id fcc9b182716c4dcd335ec7d42a6c73c11092c336c68cd6b7f288290808d2e406
Jul 30 04:43:15.329821 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:15.329786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" event={"ID":"f9407b4b-76fd-47a1-8f15-718ca5d4f993","Type":"ContainerStarted","Data":"fcc9b182716c4dcd335ec7d42a6c73c11092c336c68cd6b7f288290808d2e406"}
Jul 30 04:43:18.354451 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:18.354418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" event={"ID":"f9407b4b-76fd-47a1-8f15-718ca5d4f993","Type":"ContainerStarted","Data":"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"}
Jul 30 04:43:18.354890 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:18.354484    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:18.370120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:18.370028    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" podStartSLOduration=2.183591919 podStartE2EDuration="4.370011196s" totalImagesPullingTime="2.186419277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:43:18.369674162 +0000 UTC m=+3491.186610639" watchObservedRunningTime="2026-07-30 04:43:18.370011196 +0000 UTC m=+3491.186947676"
Jul 30 04:43:21.638973 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:21.638937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:43:24.365202 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:24.365172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:43:39.687691 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.687601    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:43:39.692814 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.692794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.695923 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.695899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 30 04:43:39.696617 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.696598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-eda6fc-openshift-default-dockercfg-j7snj\""
Jul 30 04:43:39.709715 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.709693    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:43:39.854471 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854550    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854835 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2xw\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854835 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.854835 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.854714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.955843 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.955843 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.955843 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.955843 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2xw\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.955966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956389 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956389 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956389 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956389 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.956939 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.956916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.958258 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.958235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.958385 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.958323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.963516 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.963498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:39.963687 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:39.963663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2xw\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw\") pod \"e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") " pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:40.003653 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:40.003634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:40.140763 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:40.140720    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:43:40.143026 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:43:40.143000    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc002a8a9_c810_4f7d_903c_47008448fd39.slice/crio-61fcb451bfcab866f0a9ad0130c4acc99dbc9cbc787a8e9e88485bfa81f062e8 WatchSource:0}: Error finding container 61fcb451bfcab866f0a9ad0130c4acc99dbc9cbc787a8e9e88485bfa81f062e8: Status 404 returned error can't find the container with id 61fcb451bfcab866f0a9ad0130c4acc99dbc9cbc787a8e9e88485bfa81f062e8
Jul 30 04:43:40.465712 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:40.465682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerStarted","Data":"61fcb451bfcab866f0a9ad0130c4acc99dbc9cbc787a8e9e88485bfa81f062e8"}
Jul 30 04:43:42.620770 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:42.620734    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:43:42.621053 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:42.620823    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:43:42.621053 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:42.620852    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:43:43.481700 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:43.481666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerStarted","Data":"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"}
Jul 30 04:43:43.501123 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:43.501072    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podStartSLOduration=2.02608633 podStartE2EDuration="4.501059095s" totalImagesPullingTime="2.474972765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:43:43.500207813 +0000 UTC m=+3516.317144307" watchObservedRunningTime="2026-07-30 04:43:43.501059095 +0000 UTC m=+3516.317995570"
Jul 30 04:43:44.004514 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:44.004482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:44.009265 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:44.009242    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:44.486888 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:44.486854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:44.487750 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:44.487727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:43:46.521078 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.521050    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:43:46.524544 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.524511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.526934 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.526911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-eda6fc-serving-cert\""
Jul 30 04:43:46.536116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.536094    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:43:46.613604 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.613581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.613709 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.613616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmmw\" (UniqueName: \"kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.613752 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.613705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.714839 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.714815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.714964 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.714864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmmw\" (UniqueName: \"kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.714964 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.714917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.715090 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:46.715063    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-serving-cert: secret "maas-api-e2e-ipp-b-eda6fc-serving-cert" not found
Jul 30 04:43:46.715144 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:43:46.715129    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls podName:a01c7e23-e6d1-4dd9-8d4c-49c42dc73997 nodeName:}" failed. No retries permitted until 2026-07-30 04:43:47.215108998 +0000 UTC m=+3520.032045471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls") pod "maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" (UID: "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997") : secret "maas-api-e2e-ipp-b-eda6fc-serving-cert" not found
Jul 30 04:43:46.715403 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.715385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:46.722769 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:46.722749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmmw\" (UniqueName: \"kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:47.218285 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:47.218255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:47.220594 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:47.220563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") pod \"maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:47.437596 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:47.437561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:47.574370 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:47.574345    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:43:47.575788 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:43:47.575761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01c7e23_e6d1_4dd9_8d4c_49c42dc73997.slice/crio-9b2ead354d432deda4440f2a6c963aef7bc4625c8c1f8a9b3d7f8212b6de0724 WatchSource:0}: Error finding container 9b2ead354d432deda4440f2a6c963aef7bc4625c8c1f8a9b3d7f8212b6de0724: Status 404 returned error can't find the container with id 9b2ead354d432deda4440f2a6c963aef7bc4625c8c1f8a9b3d7f8212b6de0724
Jul 30 04:43:48.516744 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:48.516710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" event={"ID":"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997","Type":"ContainerStarted","Data":"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"}
Jul 30 04:43:48.516937 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:48.516855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:48.516937 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:48.516890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" event={"ID":"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997","Type":"ContainerStarted","Data":"9b2ead354d432deda4440f2a6c963aef7bc4625c8c1f8a9b3d7f8212b6de0724"}
Jul 30 04:43:48.532960 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:48.532923    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" podStartSLOduration=2.220330043 podStartE2EDuration="2.532911874s" totalImagesPullingTime="312.581831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:43:48.531493485 +0000 UTC m=+3521.348429962" watchObservedRunningTime="2026-07-30 04:43:48.532911874 +0000 UTC m=+3521.349848350"
Jul 30 04:43:51.255102 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:51.255071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:43:54.527410 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:54.527376    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:43:59.488395 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.488358    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 30 04:43:59.488841 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.488433    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 30 04:43:59.571347 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.571316    2562 generic.go:356] "Generic (PLEG): container finished" podID="c002a8a9-c810-4f7d-903c-47008448fd39" containerID="6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8" exitCode=0
Jul 30 04:43:59.571486 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.571392    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerDied","Data":"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"}
Jul 30 04:43:59.571779 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.571762    2562 scope.go:117] "RemoveContainer" containerID="6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"
Jul 30 04:43:59.573718 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.573688    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:43:59.573782 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.573753    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:43:59.573828 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:43:59.573782    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:44:00.578465 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:00.578433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerStarted","Data":"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"}
Jul 30 04:44:01.004390 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:01.004355    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:44:01.004606 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:01.004475    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 30 04:44:01.004606 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:01.004541    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 30 04:44:02.005044 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:02.004999    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 30 04:44:02.005565 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:02.005064    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 30 04:44:03.004171 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:03.004127    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 30 04:44:03.004367 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:03.004200    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 30 04:44:04.005053 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:04.005019    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 30 04:44:04.005534 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:04.005079    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 30 04:44:05.007912 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:05.007882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:44:05.008365 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:05.008237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:44:05.008967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:05.008942    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:44:27.947816 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.947781    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:44:27.951265 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.951245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:27.954581 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.954561    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a1808a\"/\"kube-root-ca.crt\""
Jul 30 04:44:27.954581 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.954571    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a1808a\"/\"default-dockercfg-qqq88\""
Jul 30 04:44:27.954737 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.954650    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a1808a\"/\"ipp-route-a1808a-kserve-self-signed-certs\""
Jul 30 04:44:27.954737 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.954657    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a1808a\"/\"openshift-service-ca.crt\""
Jul 30 04:44:27.960829 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:27.960810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:44:28.040852 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.040829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.040990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.040881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.040990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.040910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.040990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.040950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.040990 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.040980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.041160 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.041006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst77\" (UniqueName: \"kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141534 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst77\" (UniqueName: \"kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141688 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141688 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141688 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141938 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.141938 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.141716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.142059 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.142035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.142117 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.142067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.142117 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.142088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.143985 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.143958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.144252 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.144231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.149782 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.149764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst77\" (UniqueName: \"kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77\") pod \"ipp-route-a1808a-kserve-77d57dcf59-xgjtl\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") " pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.261113 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.261082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:28.389530 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:44:28.389495    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc5ce01_2bfe_4f14_ac87_fb87f071b77d.slice/crio-40d4ff309cd851e0b4b07ea279e35bad48149a079fdb17eaf5022a5d79264e11 WatchSource:0}: Error finding container 40d4ff309cd851e0b4b07ea279e35bad48149a079fdb17eaf5022a5d79264e11: Status 404 returned error can't find the container with id 40d4ff309cd851e0b4b07ea279e35bad48149a079fdb17eaf5022a5d79264e11
Jul 30 04:44:28.390449 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.390420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:44:28.727642 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.727610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" event={"ID":"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d","Type":"ContainerStarted","Data":"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"}
Jul 30 04:44:28.727793 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.727647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" event={"ID":"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d","Type":"ContainerStarted","Data":"40d4ff309cd851e0b4b07ea279e35bad48149a079fdb17eaf5022a5d79264e11"}
Jul 30 04:44:28.744402 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:28.744360    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" podStartSLOduration=1.744346258 podStartE2EDuration="1.744346258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:44:28.74376155 +0000 UTC m=+3561.560698026" watchObservedRunningTime="2026-07-30 04:44:28.744346258 +0000 UTC m=+3561.561282734"
Jul 30 04:44:32.037583 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:32.037545    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:44:38.262288 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:38.262252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:38.262686 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:38.262309    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:38.274233 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:38.274211    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:38.787544 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:38.787497    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:44:57.546045 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:44:57.546008    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:45:00.147501 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.147419    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"]
Jul 30 04:45:00.150940 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.150923    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"]
Jul 30 04:45:00.151047 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.151029    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"
Jul 30 04:45:00.153355 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.153334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:45:00.154325 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.154305    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"]
Jul 30 04:45:00.154464 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.154441    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"
Jul 30 04:45:00.157898 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.157879    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"]
Jul 30 04:45:00.158075 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.157992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"
Jul 30 04:45:00.158259 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.158237    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"]
Jul 30 04:45:00.166471 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.166454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"
Jul 30 04:45:00.171924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.171797    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"]
Jul 30 04:45:00.175088 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.175071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"
Jul 30 04:45:00.207046 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.207023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"
Jul 30 04:45:00.325242 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.325213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"]
Jul 30 04:45:00.327199 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:45:00.327169    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdda73ea_b589_4a8a_9d44_00e51eceffa5.slice/crio-74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d WatchSource:0}: Error finding container 74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d: Status 404 returned error can't find the container with id 74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d
Jul 30 04:45:00.383054 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.383028    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"]
Jul 30 04:45:00.384562 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:45:00.384516    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed0f9ec_6c04_40a8_9640_ca68769db3dd.slice/crio-9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e WatchSource:0}: Error finding container 9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e: Status 404 returned error can't find the container with id 9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e
Jul 30 04:45:00.555587 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.555564    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"]
Jul 30 04:45:00.588607 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:45:00.588578    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40bd02d_d823_4115_bd62_0770141aed51.slice/crio-8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545 WatchSource:0}: Error finding container 8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545: Status 404 returned error can't find the container with id 8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545
Jul 30 04:45:00.892712 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.892674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8" event={"ID":"a40bd02d-d823-4115-bd62-0770141aed51","Type":"ContainerStarted","Data":"8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545"}
Jul 30 04:45:00.894484 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.894459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm" event={"ID":"cdda73ea-b589-4a8a-9d44-00e51eceffa5","Type":"ContainerStarted","Data":"53fc3eea220c06132865f5015853b0c2140da17b6a1220ff8a2ca008a8a92760"}
Jul 30 04:45:00.894687 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.894668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm" event={"ID":"cdda73ea-b589-4a8a-9d44-00e51eceffa5","Type":"ContainerStarted","Data":"74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d"}
Jul 30 04:45:00.896387 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.896363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p" event={"ID":"fed0f9ec-6c04-40a8-9640-ca68769db3dd","Type":"ContainerStarted","Data":"7e0d4c42ea286cbad46c14a5a18ee5024cfb8005adf76186f5ef7ec171d2a7bc"}
Jul 30 04:45:00.896481 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.896412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p" event={"ID":"fed0f9ec-6c04-40a8-9640-ca68769db3dd","Type":"ContainerStarted","Data":"9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e"}
Jul 30 04:45:00.909205 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.909157    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm" podStartSLOduration=0.909128344 podStartE2EDuration="909.128344ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:00.908112728 +0000 UTC m=+3593.725049206" watchObservedRunningTime="2026-07-30 04:45:00.909128344 +0000 UTC m=+3593.726064821"
Jul 30 04:45:00.922160 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:00.922119    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p" podStartSLOduration=0.922107445 podStartE2EDuration="922.107445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:00.920397635 +0000 UTC m=+3593.737334135" watchObservedRunningTime="2026-07-30 04:45:00.922107445 +0000 UTC m=+3593.739043912"
Jul 30 04:45:01.902770 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.902740    2562 generic.go:356] "Generic (PLEG): container finished" podID="a40bd02d-d823-4115-bd62-0770141aed51" containerID="513a4990c5532ef3f3547c3159cd9d9fb8843d9736f52f05de87bb62790165a3" exitCode=0
Jul 30 04:45:01.903130 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.902824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8" event={"ID":"a40bd02d-d823-4115-bd62-0770141aed51","Type":"ContainerDied","Data":"513a4990c5532ef3f3547c3159cd9d9fb8843d9736f52f05de87bb62790165a3"}
Jul 30 04:45:01.904286 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.904264    2562 generic.go:356] "Generic (PLEG): container finished" podID="cdda73ea-b589-4a8a-9d44-00e51eceffa5" containerID="53fc3eea220c06132865f5015853b0c2140da17b6a1220ff8a2ca008a8a92760" exitCode=0
Jul 30 04:45:01.904402 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.904339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm" event={"ID":"cdda73ea-b589-4a8a-9d44-00e51eceffa5","Type":"ContainerDied","Data":"53fc3eea220c06132865f5015853b0c2140da17b6a1220ff8a2ca008a8a92760"}
Jul 30 04:45:01.905786 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.905764    2562 generic.go:356] "Generic (PLEG): container finished" podID="fed0f9ec-6c04-40a8-9640-ca68769db3dd" containerID="7e0d4c42ea286cbad46c14a5a18ee5024cfb8005adf76186f5ef7ec171d2a7bc" exitCode=0
Jul 30 04:45:01.905899 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:01.905849    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p" event={"ID":"fed0f9ec-6c04-40a8-9640-ca68769db3dd","Type":"ContainerDied","Data":"7e0d4c42ea286cbad46c14a5a18ee5024cfb8005adf76186f5ef7ec171d2a7bc"}
Jul 30 04:45:03.041995 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.041962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"
Jul 30 04:45:03.093639 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.093612    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"
Jul 30 04:45:03.096651 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.096624    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"
Jul 30 04:45:03.916905 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.916880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"
Jul 30 04:45:03.917037 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.916917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p" event={"ID":"fed0f9ec-6c04-40a8-9640-ca68769db3dd","Type":"ContainerDied","Data":"9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e"}
Jul 30 04:45:03.917037 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.916962    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c11d0cecc51533fbab30b8964c4e12f42409c5e044d46ff0520e7406cf57b1e"
Jul 30 04:45:03.918151 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.918133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8" event={"ID":"a40bd02d-d823-4115-bd62-0770141aed51","Type":"ContainerDied","Data":"8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545"}
Jul 30 04:45:03.918151 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.918143    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"
Jul 30 04:45:03.918315 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.918166    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8662ba38ff4f413a8750b359163b391f7d4c41537d461d190bee86b509ca2545"
Jul 30 04:45:03.919499 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.919482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm" event={"ID":"cdda73ea-b589-4a8a-9d44-00e51eceffa5","Type":"ContainerDied","Data":"74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d"}
Jul 30 04:45:03.919602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.919507    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"
Jul 30 04:45:03.919602 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:03.919512    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fc128d219814385e2a8a0d0640d1a7926b95e0021df999fbbdca960c06237d"
Jul 30 04:45:10.257568 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.257531    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:10.258138 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258116    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed0f9ec-6c04-40a8-9640-ca68769db3dd" containerName="cleanup"
Jul 30 04:45:10.258196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258142    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed0f9ec-6c04-40a8-9640-ca68769db3dd" containerName="cleanup"
Jul 30 04:45:10.258196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258165    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdda73ea-b589-4a8a-9d44-00e51eceffa5" containerName="cleanup"
Jul 30 04:45:10.258196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258175    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdda73ea-b589-4a8a-9d44-00e51eceffa5" containerName="cleanup"
Jul 30 04:45:10.258305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258195    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40bd02d-d823-4115-bd62-0770141aed51" containerName="cleanup"
Jul 30 04:45:10.258305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258204    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40bd02d-d823-4115-bd62-0770141aed51" containerName="cleanup"
Jul 30 04:45:10.258305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258302    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdda73ea-b589-4a8a-9d44-00e51eceffa5" containerName="cleanup"
Jul 30 04:45:10.258406 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258312    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed0f9ec-6c04-40a8-9640-ca68769db3dd" containerName="cleanup"
Jul 30 04:45:10.258406 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.258318    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40bd02d-d823-4115-bd62-0770141aed51" containerName="cleanup"
Jul 30 04:45:10.261438 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.261422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.263742 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.263724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1ddcb8-openshift-default-dockercfg-fwczb\""
Jul 30 04:45:10.273977 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.273955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:10.276715 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.276842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.276842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.276842 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.277034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.277034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.277034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.277034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.276988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbdl\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.277034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.277022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378466 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbdl\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378690 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.378967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.379116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.378964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.379116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.379001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.379116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.379029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.379116 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.379059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.379549 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.379510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.381100 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.381079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.381233 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.381217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.386675 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.386654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.386865 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.386838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbdl\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl\") pod \"e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") " pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.571974 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.571882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:10.736112 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.736082    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:10.738091 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:45:10.738062    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a326971_b71a_4eef_8040_a64b9d114781.slice/crio-df2e69ea7bb7ce4f2ac2d9e3ef25748593672014381b34b9145ca7fc925af386 WatchSource:0}: Error finding container df2e69ea7bb7ce4f2ac2d9e3ef25748593672014381b34b9145ca7fc925af386: Status 404 returned error can't find the container with id df2e69ea7bb7ce4f2ac2d9e3ef25748593672014381b34b9145ca7fc925af386
Jul 30 04:45:10.740833 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.740796    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:10.740950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.740862    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:10.740950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.740890    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:10.958174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.958080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerStarted","Data":"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"}
Jul 30 04:45:10.958174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.958141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerStarted","Data":"df2e69ea7bb7ce4f2ac2d9e3ef25748593672014381b34b9145ca7fc925af386"}
Jul 30 04:45:10.977193 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:10.977141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podStartSLOduration=0.977124531 podStartE2EDuration="977.124531ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:45:10.975454883 +0000 UTC m=+3603.792391360" watchObservedRunningTime="2026-07-30 04:45:10.977124531 +0000 UTC m=+3603.794061007"
Jul 30 04:45:11.572097 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:11.572061    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:11.577251 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:11.577225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:11.963791 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:11.963700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:11.964631 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:11.964612    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:21.243900 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:21.243865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:45:28.005683 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:28.005634    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 30 04:45:28.006093 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:28.005700    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 30 04:45:29.050655 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.050578    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a326971-b71a-4eef-8040-a64b9d114781" containerID="3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53" exitCode=0
Jul 30 04:45:29.051047 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.050649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerDied","Data":"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"}
Jul 30 04:45:29.051047 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.050997    2562 scope.go:117] "RemoveContainer" containerID="3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"
Jul 30 04:45:29.054699 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.054667    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:29.054818 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.054739    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:29.054818 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:29.054770    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 30 04:45:30.057869 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:30.057833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerStarted","Data":"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"}
Jul 30 04:45:30.572898 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:30.572865    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:30.573114 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:30.573032    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 30 04:45:30.573114 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:30.573075    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 30 04:45:31.572916 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:31.572883    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 30 04:45:31.573299 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:31.572938    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 30 04:45:32.573045 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:32.573008    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 30 04:45:32.573425 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:32.573077    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 30 04:45:33.572641 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:33.572602    2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 30 04:45:33.572828 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:33.572668    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 30 04:45:34.576568 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:34.576539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:34.577043 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:34.576908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:34.577603 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:34.577583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:48.741484 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:48.741451    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:45:51.280540 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:51.280496    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:51.281007 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:51.280701    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy" containerID="cri-o://9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43" gracePeriod=30
Jul 30 04:45:56.524595 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.524572    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:56.647825 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.647825 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.647825 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647823    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzbdl\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647908    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.647955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648014    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648103 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket\") pod \"8a326971-b71a-4eef-8040-a64b9d114781\" (UID: \"8a326971-b71a-4eef-8040-a64b9d114781\") "
Jul 30 04:45:56.648390 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648261    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:45:56.648390 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648270    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:45:56.648496 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:45:56.648496 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:45:56.648700 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.648679    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:45:56.650147 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.650125    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl" (OuterVolumeSpecName: "kube-api-access-nzbdl") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "kube-api-access-nzbdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:45:56.650268 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.650200    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:45:56.650410 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.650386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 04:45:56.650410 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.650402    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a326971-b71a-4eef-8040-a64b9d114781" (UID: "8a326971-b71a-4eef-8040-a64b9d114781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:45:56.749594 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749561    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-envoy\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749594 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749593    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-socket\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749594 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749602    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-istio-data\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749610    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a326971-b71a-4eef-8040-a64b9d114781-istio-podinfo\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749618    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbdl\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-kube-api-access-nzbdl\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749628    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a326971-b71a-4eef-8040-a64b9d114781-istiod-ca-cert\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749635    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a326971-b71a-4eef-8040-a64b9d114781-istio-token\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749643    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-workload-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:56.749755 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:56.749650    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a326971-b71a-4eef-8040-a64b9d114781-credential-socket\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:45:57.194598 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.194564    2562 generic.go:356] "Generic (PLEG): container finished" podID="8a326971-b71a-4eef-8040-a64b9d114781" containerID="9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43" exitCode=0
Jul 30 04:45:57.194803 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.194620    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"
Jul 30 04:45:57.194803 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.194646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerDied","Data":"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"}
Jul 30 04:45:57.194803 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.194700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t" event={"ID":"8a326971-b71a-4eef-8040-a64b9d114781","Type":"ContainerDied","Data":"df2e69ea7bb7ce4f2ac2d9e3ef25748593672014381b34b9145ca7fc925af386"}
Jul 30 04:45:57.194803 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.194722    2562 scope.go:117] "RemoveContainer" containerID="9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"
Jul 30 04:45:57.206877 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.206862    2562 scope.go:117] "RemoveContainer" containerID="3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"
Jul 30 04:45:57.215167 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.215152    2562 scope.go:117] "RemoveContainer" containerID="9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"
Jul 30 04:45:57.215410 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:45:57.215393    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43\": container with ID starting with 9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43 not found: ID does not exist" containerID="9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"
Jul 30 04:45:57.215468 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.215417    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43"} err="failed to get container status \"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43\": rpc error: code = NotFound desc = could not find container \"9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43\": container with ID starting with 9e3199ff0e29aa1a5067a8ee262daeacfab71f84c21281af7ef2a22031c79f43 not found: ID does not exist"
Jul 30 04:45:57.215468 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.215433    2562 scope.go:117] "RemoveContainer" containerID="3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"
Jul 30 04:45:57.215669 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:45:57.215645    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53\": container with ID starting with 3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53 not found: ID does not exist" containerID="3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"
Jul 30 04:45:57.215712 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.215679    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53"} err="failed to get container status \"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53\": rpc error: code = NotFound desc = could not find container \"3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53\": container with ID starting with 3e382884a0226481e8fa4edb0958c9b02ffb0715f0bb2fe4e6817092247e7d53 not found: ID does not exist"
Jul 30 04:45:57.215955 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.215940    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:57.218923 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.218904    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ddcb8-openshift-default-7d8c5749dd-xtl5t"]
Jul 30 04:45:57.792529 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:45:57.792481    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a326971-b71a-4eef-8040-a64b9d114781" path="/var/lib/kubelet/pods/8a326971-b71a-4eef-8040-a64b9d114781/volumes"
Jul 30 04:46:10.247731 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.247692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:46:10.481022 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.480990    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:46:10.481249 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.481217    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" podUID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" containerName="maas-api" containerID="cri-o://f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d" gracePeriod=30
Jul 30 04:46:10.501157 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.501092    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"]
Jul 30 04:46:10.502864 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.502742    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1808a-29756445-gj9b8"]
Jul 30 04:46:10.730439 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.730413    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:46:10.858706 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.858633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls\") pod \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") "
Jul 30 04:46:10.858706 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.858672    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca\") pod \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") "
Jul 30 04:46:10.858924 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.858750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz4v4\" (UniqueName: \"kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4\") pod \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\" (UID: \"f9407b4b-76fd-47a1-8f15-718ca5d4f993\") "
Jul 30 04:46:10.859068 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.859043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9407b4b-76fd-47a1-8f15-718ca5d4f993" (UID: "f9407b4b-76fd-47a1-8f15-718ca5d4f993"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:46:10.860886 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.860854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4" (OuterVolumeSpecName: "kube-api-access-tz4v4") pod "f9407b4b-76fd-47a1-8f15-718ca5d4f993" (UID: "f9407b4b-76fd-47a1-8f15-718ca5d4f993"). InnerVolumeSpecName "kube-api-access-tz4v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:46:10.860970 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.860892    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9407b4b-76fd-47a1-8f15-718ca5d4f993" (UID: "f9407b4b-76fd-47a1-8f15-718ca5d4f993"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:46:10.959906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.959879    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4v4\" (UniqueName: \"kubernetes.io/projected/f9407b4b-76fd-47a1-8f15-718ca5d4f993-kube-api-access-tz4v4\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:10.959906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.959905    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9407b4b-76fd-47a1-8f15-718ca5d4f993-maas-api-tls\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:10.960053 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:10.959914    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9407b4b-76fd-47a1-8f15-718ca5d4f993-openshift-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:11.272572 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.272517    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" containerID="f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d" exitCode=0
Jul 30 04:46:11.272994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.272582    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"
Jul 30 04:46:11.272994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.272581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" event={"ID":"f9407b4b-76fd-47a1-8f15-718ca5d4f993","Type":"ContainerDied","Data":"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"}
Jul 30 04:46:11.272994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.272714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42" event={"ID":"f9407b4b-76fd-47a1-8f15-718ca5d4f993","Type":"ContainerDied","Data":"fcc9b182716c4dcd335ec7d42a6c73c11092c336c68cd6b7f288290808d2e406"}
Jul 30 04:46:11.272994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.272748    2562 scope.go:117] "RemoveContainer" containerID="f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"
Jul 30 04:46:11.281814 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.281790    2562 scope.go:117] "RemoveContainer" containerID="f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"
Jul 30 04:46:11.282053 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:46:11.282033    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d\": container with ID starting with f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d not found: ID does not exist" containerID="f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"
Jul 30 04:46:11.282090 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.282062    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d"} err="failed to get container status \"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d\": rpc error: code = NotFound desc = could not find container \"f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d\": container with ID starting with f1ddc6fd953d1dab9112a835f33755f5e20b41212f3a06e1bf714b953a23206d not found: ID does not exist"
Jul 30 04:46:11.293740 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.293719    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:46:11.297047 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.297027    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a1808a-68df456968-gbt42"]
Jul 30 04:46:11.785258 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.785231    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a40bd02d-d823-4115-bd62-0770141aed51" path="/var/lib/kubelet/pods/a40bd02d-d823-4115-bd62-0770141aed51/volumes"
Jul 30 04:46:11.785592 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:11.785577    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" path="/var/lib/kubelet/pods/f9407b4b-76fd-47a1-8f15-718ca5d4f993/volumes"
Jul 30 04:46:16.045604 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:16.045570    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:46:20.647783 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:20.647745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:46:31.064771 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.064640    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" podUID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" containerName="main" containerID="cri-o://9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96" gracePeriod=45
Jul 30 04:46:31.317034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.316983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:46:31.385827 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.385791    2562 generic.go:356] "Generic (PLEG): container finished" podID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" containerID="9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96" exitCode=0
Jul 30 04:46:31.385991 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.385834    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"
Jul 30 04:46:31.385991 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.385875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" event={"ID":"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d","Type":"ContainerDied","Data":"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"}
Jul 30 04:46:31.385991 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.385931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl" event={"ID":"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d","Type":"ContainerDied","Data":"40d4ff309cd851e0b4b07ea279e35bad48149a079fdb17eaf5022a5d79264e11"}
Jul 30 04:46:31.385991 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.385955    2562 scope.go:117] "RemoveContainer" containerID="9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"
Jul 30 04:46:31.393756 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.393739    2562 scope.go:117] "RemoveContainer" containerID="9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"
Jul 30 04:46:31.393992 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:46:31.393976    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96\": container with ID starting with 9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96 not found: ID does not exist" containerID="9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"
Jul 30 04:46:31.394041 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.394000    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96"} err="failed to get container status \"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96\": rpc error: code = NotFound desc = could not find container \"9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96\": container with ID starting with 9f41e7c56c87ed3c9186b40c440f9d50aafeabc4f83ea2c88de9054206627c96 not found: ID does not exist"
Jul 30 04:46:31.418306 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418286    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418310    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418386 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418336    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kst77\" (UniqueName: \"kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418383    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418434    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418586 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs\") pod \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\" (UID: \"dbc5ce01-2bfe-4f14-ac87-fb87f071b77d\") "
Jul 30 04:46:31.418644 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:46:31.418644 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:46:31.418761 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home" (OuterVolumeSpecName: "home") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:46:31.418874 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418858    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-model-cache\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.418928 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418880    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tmp-dir\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.418928 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.418892    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-home\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.420511 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.420487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm" (OuterVolumeSpecName: "dshm") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:46:31.420511 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.420503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77" (OuterVolumeSpecName: "kube-api-access-kst77") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "kube-api-access-kst77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:46:31.420665 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.420538    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" (UID: "dbc5ce01-2bfe-4f14-ac87-fb87f071b77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:46:31.520153 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.520130    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-tls-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.520153 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.520151    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst77\" (UniqueName: \"kubernetes.io/projected/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-kube-api-access-kst77\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.520286 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.520159    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d-dshm\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:46:31.707014 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.706982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:46:31.712784 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.712760    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a1808a/ipp-route-a1808a-kserve-77d57dcf59-xgjtl"]
Jul 30 04:46:31.785635 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:31.785608    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" path="/var/lib/kubelet/pods/dbc5ce01-2bfe-4f14-ac87-fb87f071b77d/volumes"
Jul 30 04:46:59.069492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069460    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"]
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069833    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069846    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" containerName="maas-api"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069878    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" containerName="maas-api"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069885    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" containerName="main"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069891    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" containerName="main"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069945    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:46:59.069950 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069955    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9407b4b-76fd-47a1-8f15-718ca5d4f993" containerName="maas-api"
Jul 30 04:46:59.070219 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.069962    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc5ce01-2bfe-4f14-ac87-fb87f071b77d" containerName="main"
Jul 30 04:46:59.073127 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.073112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:46:59.075453 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.075434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:46:59.079458 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.079437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"]
Jul 30 04:46:59.123835 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.123812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c\" (UID: \"bb554f4b-41cb-4537-90a4-c5f770747511\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:46:59.224588 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.224563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c\" (UID: \"bb554f4b-41cb-4537-90a4-c5f770747511\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:46:59.225176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.225156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c\" (UID: \"bb554f4b-41cb-4537-90a4-c5f770747511\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:46:59.384406 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.384349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:46:59.716344 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:46:59.716321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"]
Jul 30 04:46:59.718118 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:46:59.718091    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb554f4b_41cb_4537_90a4_c5f770747511.slice/crio-c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5 WatchSource:0}: Error finding container c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5: Status 404 returned error can't find the container with id c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5
Jul 30 04:47:00.534564 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:00.534506    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb554f4b-41cb-4537-90a4-c5f770747511" containerID="1d3615ae99e745ca33b261812e02b5dcc293e4f5f4dc561dbf3c209d25918b44" exitCode=0
Jul 30 04:47:00.534956 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:00.534580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c" event={"ID":"bb554f4b-41cb-4537-90a4-c5f770747511","Type":"ContainerDied","Data":"1d3615ae99e745ca33b261812e02b5dcc293e4f5f4dc561dbf3c209d25918b44"}
Jul 30 04:47:00.534956 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:00.534649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c" event={"ID":"bb554f4b-41cb-4537-90a4-c5f770747511","Type":"ContainerStarted","Data":"c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5"}
Jul 30 04:47:01.673308 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:01.673284    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:47:01.744993 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:01.744970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca\") pod \"bb554f4b-41cb-4537-90a4-c5f770747511\" (UID: \"bb554f4b-41cb-4537-90a4-c5f770747511\") "
Jul 30 04:47:01.745337 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:01.745318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb554f4b-41cb-4537-90a4-c5f770747511" (UID: "bb554f4b-41cb-4537-90a4-c5f770747511"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:47:01.845995 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:01.845970    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb554f4b-41cb-4537-90a4-c5f770747511-maas-api-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:02.545599 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:02.545564    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"
Jul 30 04:47:02.545599 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:02.545576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c" event={"ID":"bb554f4b-41cb-4537-90a4-c5f770747511","Type":"ContainerDied","Data":"c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5"}
Jul 30 04:47:02.545825 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:02.545682    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c949ac09eaafae571b246d0d73cf949dd59b5cdcfec1a4c18270c8f9211200d5"
Jul 30 04:47:09.488870 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.488823    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:47:09.489387 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.489069    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerName="maas-api" containerID="cri-o://c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb" gracePeriod=30
Jul 30 04:47:09.518658 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.518627    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"]
Jul 30 04:47:09.520983 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.520958    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eda6fc-29756445-vsrjm"]
Jul 30 04:47:09.522059 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.522029    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.66:8443/health\": dial tcp 10.133.0.66:8443: connect: connection refused"
Jul 30 04:47:09.699512 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.699479    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"]
Jul 30 04:47:09.702304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.702277    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eda6fc-8kx2c"]
Jul 30 04:47:09.736087 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.736064    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:47:09.785693 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.785619    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb554f4b-41cb-4537-90a4-c5f770747511" path="/var/lib/kubelet/pods/bb554f4b-41cb-4537-90a4-c5f770747511/volumes"
Jul 30 04:47:09.785998 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.785979    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdda73ea-b589-4a8a-9d44-00e51eceffa5" path="/var/lib/kubelet/pods/cdda73ea-b589-4a8a-9d44-00e51eceffa5/volumes"
Jul 30 04:47:09.803645 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.803622    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca\") pod \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") "
Jul 30 04:47:09.803751 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.803666    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgmmw\" (UniqueName: \"kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw\") pod \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") "
Jul 30 04:47:09.803751 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.803687    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") pod \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\" (UID: \"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997\") "
Jul 30 04:47:09.804020 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.803998    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" (UID: "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:47:09.805733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.805713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" (UID: "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 04:47:09.805733 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.805725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw" (OuterVolumeSpecName: "kube-api-access-qgmmw") pod "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" (UID: "a01c7e23-e6d1-4dd9-8d4c-49c42dc73997"). InnerVolumeSpecName "kube-api-access-qgmmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:47:09.905161 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.905131    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-openshift-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:09.905161 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.905161    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmmw\" (UniqueName: \"kubernetes.io/projected/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-kube-api-access-qgmmw\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:09.905305 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:09.905176    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997-maas-api-tls\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:10.589504 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.589473    2562 generic.go:356] "Generic (PLEG): container finished" podID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerID="c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb" exitCode=0
Jul 30 04:47:10.589994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.589532    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"
Jul 30 04:47:10.589994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.589574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" event={"ID":"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997","Type":"ContainerDied","Data":"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"}
Jul 30 04:47:10.589994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.589636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744" event={"ID":"a01c7e23-e6d1-4dd9-8d4c-49c42dc73997","Type":"ContainerDied","Data":"9b2ead354d432deda4440f2a6c963aef7bc4625c8c1f8a9b3d7f8212b6de0724"}
Jul 30 04:47:10.589994 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.589669    2562 scope.go:117] "RemoveContainer" containerID="c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"
Jul 30 04:47:10.599905 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.599886    2562 scope.go:117] "RemoveContainer" containerID="c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"
Jul 30 04:47:10.600181 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:47:10.600163    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb\": container with ID starting with c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb not found: ID does not exist" containerID="c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"
Jul 30 04:47:10.600279 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.600188    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb"} err="failed to get container status \"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb\": rpc error: code = NotFound desc = could not find container \"c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb\": container with ID starting with c9ac861d5130f231308b9ec80fb59af3cd3e3c0ed0ec96bf6042399cff5712cb not found: ID does not exist"
Jul 30 04:47:10.610269 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.610247    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:47:10.611947 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:10.611928    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-eda6fc-6c4ddd4ff9-zj744"]
Jul 30 04:47:11.785015 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:11.784976    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" path="/var/lib/kubelet/pods/a01c7e23-e6d1-4dd9-8d4c-49c42dc73997/volumes"
Jul 30 04:47:12.647026 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:12.646994    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:47:16.127694 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.127653    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:47:16.128142 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.127900    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" containerID="cri-o://06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd" gracePeriod=30
Jul 30 04:47:16.519891 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.519862    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"]
Jul 30 04:47:16.520278 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520262    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb554f4b-41cb-4537-90a4-c5f770747511" containerName="revoke-keys"
Jul 30 04:47:16.520324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520280    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb554f4b-41cb-4537-90a4-c5f770747511" containerName="revoke-keys"
Jul 30 04:47:16.520324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520299    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerName="maas-api"
Jul 30 04:47:16.520324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520304    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerName="maas-api"
Jul 30 04:47:16.520324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520312    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:47:16.520324 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520317    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:47:16.520475 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520376    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01c7e23-e6d1-4dd9-8d4c-49c42dc73997" containerName="maas-api"
Jul 30 04:47:16.520475 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520383    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb554f4b-41cb-4537-90a4-c5f770747511" containerName="revoke-keys"
Jul 30 04:47:16.520475 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.520392    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a326971-b71a-4eef-8040-a64b9d114781" containerName="istio-proxy"
Jul 30 04:47:16.524815 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.524799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:16.527385 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.527359    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:47:16.527544 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.527364    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 04:47:16.528902 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.528880    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"]
Jul 30 04:47:16.661740 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.661715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn\" (UID: \"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:16.762073 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.762047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn\" (UID: \"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:16.762674 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.762653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn\" (UID: \"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:16.836188 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.836137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:16.961844 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:16.961804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"]
Jul 30 04:47:16.963884 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:47:16.963852    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786662cd_1c9a_4a07_8c7d_ad3d90cb1b38.slice/crio-0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c WatchSource:0}: Error finding container 0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c: Status 404 returned error can't find the container with id 0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c
Jul 30 04:47:17.629848 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:17.629796    2562 generic.go:356] "Generic (PLEG): container finished" podID="786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" containerID="7f44f1ce0979956112f0134d2fc8dc94f93cc99b17c64f2db749bcc70d1c4cb4" exitCode=0
Jul 30 04:47:17.629848 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:17.629839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn" event={"ID":"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38","Type":"ContainerDied","Data":"7f44f1ce0979956112f0134d2fc8dc94f93cc99b17c64f2db749bcc70d1c4cb4"}
Jul 30 04:47:17.630260 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:17.629898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn" event={"ID":"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38","Type":"ContainerStarted","Data":"0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c"}
Jul 30 04:47:18.776801 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:18.776780    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:18.879102 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:18.879069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca\") pod \"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38\" (UID: \"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38\") "
Jul 30 04:47:18.879393 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:18.879369    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" (UID: "786662cd-1c9a-4a07-8c7d-ad3d90cb1b38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:47:18.980091 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:18.980070    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38-maas-api-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:19.639983 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:19.639956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn" event={"ID":"786662cd-1c9a-4a07-8c7d-ad3d90cb1b38","Type":"ContainerDied","Data":"0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c"}
Jul 30 04:47:19.640133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:19.639995    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"
Jul 30 04:47:19.640133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:19.640004    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a82aec93586481761b3b699280c4f19f3a7d5cc1b43dcfc7d98c4b47374b63c"
Jul 30 04:47:20.011158 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:20.011122    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 04:47:20.011553 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:20.011178    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 04:47:21.377528 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.377496    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:47:21.499597 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499741 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499616    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499741 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499642    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499741 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499741 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df2xw\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499741 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499996 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499996 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499996 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499867    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token\") pod \"c002a8a9-c810-4f7d-903c-47008448fd39\" (UID: \"c002a8a9-c810-4f7d-903c-47008448fd39\") "
Jul 30 04:47:21.499996 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.499885    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:47:21.500186 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500022    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:47:21.500186 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500070    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:47:21.500186 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500138    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-credential-socket\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.500186 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500156    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c002a8a9-c810-4f7d-903c-47008448fd39-istiod-ca-cert\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.500186 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500170    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-certs\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.500447 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:47:21.500612 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.500592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data" (OuterVolumeSpecName: "istio-data") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:47:21.502013 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.501984    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 04:47:21.502104 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.502026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token" (OuterVolumeSpecName: "istio-token") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:47:21.502303 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.502283    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 30 04:47:21.502369 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.502307    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw" (OuterVolumeSpecName: "kube-api-access-df2xw") pod "c002a8a9-c810-4f7d-903c-47008448fd39" (UID: "c002a8a9-c810-4f7d-903c-47008448fd39"). InnerVolumeSpecName "kube-api-access-df2xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 04:47:21.601059 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601035    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c002a8a9-c810-4f7d-903c-47008448fd39-istio-podinfo\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.601059 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601059    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2xw\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-kube-api-access-df2xw\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.601174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601069    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-envoy\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.601174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601078    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-workload-socket\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.601174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601086    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c002a8a9-c810-4f7d-903c-47008448fd39-istio-token\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.601174 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.601094    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c002a8a9-c810-4f7d-903c-47008448fd39-istio-data\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:21.651054 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.651031    2562 generic.go:356] "Generic (PLEG): container finished" podID="c002a8a9-c810-4f7d-903c-47008448fd39" containerID="06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd" exitCode=0
Jul 30 04:47:21.651152 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.651079    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"
Jul 30 04:47:21.651152 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.651115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerDied","Data":"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"}
Jul 30 04:47:21.651237 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.651164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb" event={"ID":"c002a8a9-c810-4f7d-903c-47008448fd39","Type":"ContainerDied","Data":"61fcb451bfcab866f0a9ad0130c4acc99dbc9cbc787a8e9e88485bfa81f062e8"}
Jul 30 04:47:21.651237 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.651187    2562 scope.go:117] "RemoveContainer" containerID="06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"
Jul 30 04:47:21.660026 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.660011    2562 scope.go:117] "RemoveContainer" containerID="6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"
Jul 30 04:47:21.667687 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.667672    2562 scope.go:117] "RemoveContainer" containerID="06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"
Jul 30 04:47:21.667956 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:47:21.667938    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd\": container with ID starting with 06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd not found: ID does not exist" containerID="06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"
Jul 30 04:47:21.668004 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.667963    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd"} err="failed to get container status \"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd\": rpc error: code = NotFound desc = could not find container \"06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd\": container with ID starting with 06210a41aadc13b0c7b97bf3e7e3578ed50d87234da6562e4bd7aa5e22db35cd not found: ID does not exist"
Jul 30 04:47:21.668004 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.667980    2562 scope.go:117] "RemoveContainer" containerID="6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"
Jul 30 04:47:21.668180 ip-10-0-138-44 kubenswrapper[2562]: E0730 04:47:21.668163    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8\": container with ID starting with 6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8 not found: ID does not exist" containerID="6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"
Jul 30 04:47:21.668236 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.668190    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8"} err="failed to get container status \"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8\": rpc error: code = NotFound desc = could not find container \"6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8\": container with ID starting with 6100698cfe10e927a1698bce62e03ee709044c77464d9368f6ee9e716df398b8 not found: ID does not exist"
Jul 30 04:47:21.671933 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.671912    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:47:21.675392 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.675369    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-eda6fc-openshift-default-75854db8db-j7gvb"]
Jul 30 04:47:21.785456 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:21.785403    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" path="/var/lib/kubelet/pods/c002a8a9-c810-4f7d-903c-47008448fd39/volumes"
Jul 30 04:47:27.225452 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:27.225411    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"]
Jul 30 04:47:27.227727 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:27.227698    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-324a29-gqwgn"]
Jul 30 04:47:27.785255 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:27.785223    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" path="/var/lib/kubelet/pods/786662cd-1c9a-4a07-8c7d-ad3d90cb1b38/volumes"
Jul 30 04:47:30.149790 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:30.149751    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:47:34.084745 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.084711    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"]
Jul 30 04:47:34.085133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085096    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.085133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085107    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.085133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085116    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.085133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085121    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.085133 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085131    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" containerName="revoke-keys"
Jul 30 04:47:34.085289 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085137    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" containerName="revoke-keys"
Jul 30 04:47:34.085289 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085194    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.085289 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085203    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="786662cd-1c9a-4a07-8c7d-ad3d90cb1b38" containerName="revoke-keys"
Jul 30 04:47:34.085289 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.085209    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c002a8a9-c810-4f7d-903c-47008448fd39" containerName="istio-proxy"
Jul 30 04:47:34.088556 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.088538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:34.091034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.091012    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l96nf\""
Jul 30 04:47:34.091155 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.091042    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 04:47:34.094946 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.094889    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"]
Jul 30 04:47:34.194564 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.194540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn\" (UID: \"da38d6d0-d540-49dd-aa24-846044726fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:34.295027 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.295003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn\" (UID: \"da38d6d0-d540-49dd-aa24-846044726fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:34.295534 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.295508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn\" (UID: \"da38d6d0-d540-49dd-aa24-846044726fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:34.400340 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.400276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:34.527191 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.527168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"]
Jul 30 04:47:34.529312 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:47:34.529282    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda38d6d0_d540_49dd_aa24_846044726fff.slice/crio-4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3 WatchSource:0}: Error finding container 4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3: Status 404 returned error can't find the container with id 4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3
Jul 30 04:47:34.727673 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:34.727644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn" event={"ID":"da38d6d0-d540-49dd-aa24-846044726fff","Type":"ContainerStarted","Data":"4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3"}
Jul 30 04:47:35.734176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:35.734143    2562 generic.go:356] "Generic (PLEG): container finished" podID="da38d6d0-d540-49dd-aa24-846044726fff" containerID="6300b5564a03d71bca18b5f4fc50eb7442a22ec161abedaf82e4016b80f715bb" exitCode=0
Jul 30 04:47:35.734558 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:35.734216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn" event={"ID":"da38d6d0-d540-49dd-aa24-846044726fff","Type":"ContainerDied","Data":"6300b5564a03d71bca18b5f4fc50eb7442a22ec161abedaf82e4016b80f715bb"}
Jul 30 04:47:36.874110 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:36.874087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:37.017648 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.017580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca\") pod \"da38d6d0-d540-49dd-aa24-846044726fff\" (UID: \"da38d6d0-d540-49dd-aa24-846044726fff\") "
Jul 30 04:47:37.017899 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.017877    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da38d6d0-d540-49dd-aa24-846044726fff" (UID: "da38d6d0-d540-49dd-aa24-846044726fff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 04:47:37.118235 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.118209    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da38d6d0-d540-49dd-aa24-846044726fff-maas-api-service-ca\") on node \"ip-10-0-138-44.ec2.internal\" DevicePath \"\""
Jul 30 04:47:37.745618 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.745577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn" event={"ID":"da38d6d0-d540-49dd-aa24-846044726fff","Type":"ContainerDied","Data":"4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3"}
Jul 30 04:47:37.745774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.745636    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4231826cd8ccec362cbafaee0b2e96efd21a199de022f894ef1ed39779c026a3"
Jul 30 04:47:37.745774 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:37.745599    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"
Jul 30 04:47:44.517232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:44.517194    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"]
Jul 30 04:47:44.521920 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:44.521891    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-64af9f-29756445-hv68p"]
Jul 30 04:47:44.684383 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:44.684349    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"]
Jul 30 04:47:44.686993 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:44.686963    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-64af9f-7l2rn"]
Jul 30 04:47:45.784756 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:45.784722    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da38d6d0-d540-49dd-aa24-846044726fff" path="/var/lib/kubelet/pods/da38d6d0-d540-49dd-aa24-846044726fff/volumes"
Jul 30 04:47:45.785126 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:45.784981    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed0f9ec-6c04-40a8-9640-ca68769db3dd" path="/var/lib/kubelet/pods/fed0f9ec-6c04-40a8-9640-ca68769db3dd/volumes"
Jul 30 04:47:47.246471 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:47.246437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q5bzq"]
Jul 30 04:47:51.837540 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:51.837273    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687744f755-dt7xz_25d5b1ef-16a4-465b-9ae0-2bd1491059be/authorino/0.log"
Jul 30 04:47:57.196169 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.196136    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bdxxf_03376731-166b-441e-91c4-c5d2e3152555/manager/0.log"
Jul 30 04:47:57.315234 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.315205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-52dzh_a793978b-1b44-4de8-950c-9a2d759a5a06/manager/0.log"
Jul 30 04:47:57.564651 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.564628    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2hlrv_490420af-8cae-4a85-86e2-596af1e63beb/server/0.log"
Jul 30 04:47:57.678959 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.678932    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jc2b7_81fc883b-e8d5-4b2b-b791-5a5d6daf066f/manager/0.log"
Jul 30 04:47:57.925981 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.925925    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-n9nv5_e1031033-e23b-4455-a957-7e449137dfc2/copy-manifests/0.log"
Jul 30 04:47:57.932833 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:57.932813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-n9nv5_e1031033-e23b-4455-a957-7e449137dfc2/manager/0.log"
Jul 30 04:47:58.650777 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:47:58.650750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756430-m2xct_afdaa638-fd00-485e-a88e-5e123285fb67/cleanup/0.log"
Jul 30 04:48:00.176120 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.176041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687744f755-dt7xz_25d5b1ef-16a4-465b-9ae0-2bd1491059be/authorino/0.log"
Jul 30 04:48:00.293700 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.293672    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xrjrb_7dc02ebc-a14d-4bb2-81a3-db3512a6a441/manager/0.log"
Jul 30 04:48:00.413361 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.413335    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rctsq_b843f183-8b45-4a1c-becb-4ea2dca5bdca/manager/0.log"
Jul 30 04:48:00.526176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.526151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c22nh_de6d0192-055a-438f-9845-dea2db2937a9/kuadrant-console-plugin/0.log"
Jul 30 04:48:00.652778 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.652750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8gjnc_cd8774ac-a070-4410-af23-45d6722f4f92/registry-server/0.log"
Jul 30 04:48:00.772117 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.772094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n59hc_1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8/manager/0.log"
Jul 30 04:48:00.886503 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:00.886439    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q5bzq_530be5d0-8d48-499a-976d-0df89dec736e/limitador/0.log"
Jul 30 04:48:01.002893 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:01.002867    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-psgzj_321a2104-8ee0-4d45-86af-a89aa3e9be2d/manager/0.log"
Jul 30 04:48:01.702382 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:01.702353    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58ccc58984-n82rr_10fd0823-093e-4bfc-97ac-a14dad793f8c/kube-auth-proxy/0.log"
Jul 30 04:48:02.185044 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:02.185022    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b9dc4d85-dz8bz_d1347232-d5a8-4c77-b0de-2609a00e9b05/router/0.log"
Jul 30 04:48:03.108076 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:03.108043    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-62x7v_5ba7cb01-24b8-4676-8a65-fa07a5536082/main/0.log"
Jul 30 04:48:08.270323 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.270294    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qdzjx/must-gather-hfvgw"]
Jul 30 04:48:08.270671 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.270657    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da38d6d0-d540-49dd-aa24-846044726fff" containerName="revoke-keys"
Jul 30 04:48:08.270718 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.270672    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da38d6d0-d540-49dd-aa24-846044726fff" containerName="revoke-keys"
Jul 30 04:48:08.270752 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.270739    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da38d6d0-d540-49dd-aa24-846044726fff" containerName="revoke-keys"
Jul 30 04:48:08.275052 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.275035    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.277698 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.277678    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdzjx\"/\"kube-root-ca.crt\""
Jul 30 04:48:08.277698 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.277687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdzjx\"/\"openshift-service-ca.crt\""
Jul 30 04:48:08.279084 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.278697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qdzjx\"/\"default-dockercfg-vdm2s\""
Jul 30 04:48:08.280938 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.280893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/must-gather-hfvgw"]
Jul 30 04:48:08.436801 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.436776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgpt\" (UniqueName: \"kubernetes.io/projected/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-kube-api-access-wpgpt\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.436906 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.436807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-must-gather-output\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.537575 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.537549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgpt\" (UniqueName: \"kubernetes.io/projected/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-kube-api-access-wpgpt\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.537685 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.537588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-must-gather-output\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.537864 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.537847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-must-gather-output\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.548603 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.548573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgpt\" (UniqueName: \"kubernetes.io/projected/e292b36f-79ba-4a1c-83a6-d1c68395e4f7-kube-api-access-wpgpt\") pod \"must-gather-hfvgw\" (UID: \"e292b36f-79ba-4a1c-83a6-d1c68395e4f7\") " pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.584176 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.584153    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/must-gather-hfvgw"
Jul 30 04:48:08.723573 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.723540    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/must-gather-hfvgw"]
Jul 30 04:48:08.726338 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:48:08.726308    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode292b36f_79ba_4a1c_83a6_d1c68395e4f7.slice/crio-076a86f96abb993ede52c64c40bcb9a79a5c27447a7956f96940b9fd6b2eaf75 WatchSource:0}: Error finding container 076a86f96abb993ede52c64c40bcb9a79a5c27447a7956f96940b9fd6b2eaf75: Status 404 returned error can't find the container with id 076a86f96abb993ede52c64c40bcb9a79a5c27447a7956f96940b9fd6b2eaf75
Jul 30 04:48:08.727995 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.727979    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 04:48:08.903587 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:08.903514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/must-gather-hfvgw" event={"ID":"e292b36f-79ba-4a1c-83a6-d1c68395e4f7","Type":"ContainerStarted","Data":"076a86f96abb993ede52c64c40bcb9a79a5c27447a7956f96940b9fd6b2eaf75"}
Jul 30 04:48:09.914304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:09.914207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/must-gather-hfvgw" event={"ID":"e292b36f-79ba-4a1c-83a6-d1c68395e4f7","Type":"ContainerStarted","Data":"d6926c3dc38d900ce104b4d94c41324b4349cf1f14cdbc45f410341cacfb2f73"}
Jul 30 04:48:09.914304 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:09.914265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/must-gather-hfvgw" event={"ID":"e292b36f-79ba-4a1c-83a6-d1c68395e4f7","Type":"ContainerStarted","Data":"2e3b990d44894b566ab83171338b2b92412c23fa1bc4a197ccd3155f072bbbe6"}
Jul 30 04:48:09.928918 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:09.928867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qdzjx/must-gather-hfvgw" podStartSLOduration=1.082566584 podStartE2EDuration="1.928851562s" totalImagesPullingTime="846.284978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 04:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:48:09.928479646 +0000 UTC m=+3782.745416122" watchObservedRunningTime="2026-07-30 04:48:09.928851562 +0000 UTC m=+3782.745788072"
Jul 30 04:48:11.125427 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:11.125393    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wfpd9_8a1c3a26-fa64-4f10-a63f-8c56c0912d68/global-pull-secret-syncer/0.log"
Jul 30 04:48:11.213071 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:11.213037    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v6tcn_3706bd53-387b-41be-8596-e51f050f920e/konnectivity-agent/0.log"
Jul 30 04:48:11.312439 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:11.312400    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-44.ec2.internal_28ee2cd84792d9409e5acf4090158792/haproxy/0.log"
Jul 30 04:48:15.779848 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:15.779813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687744f755-dt7xz_25d5b1ef-16a4-465b-9ae0-2bd1491059be/authorino/0.log"
Jul 30 04:48:15.819967 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:15.819930    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xrjrb_7dc02ebc-a14d-4bb2-81a3-db3512a6a441/manager/0.log"
Jul 30 04:48:15.854110 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:15.854075    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rctsq_b843f183-8b45-4a1c-becb-4ea2dca5bdca/manager/0.log"
Jul 30 04:48:15.881134 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:15.881098    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c22nh_de6d0192-055a-438f-9845-dea2db2937a9/kuadrant-console-plugin/0.log"
Jul 30 04:48:15.936596 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:15.936562    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8gjnc_cd8774ac-a070-4410-af23-45d6722f4f92/registry-server/0.log"
Jul 30 04:48:16.063470 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:16.063386    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-n59hc_1cff8f3e-dc2a-4bff-90ba-dd18404bcbd8/manager/0.log"
Jul 30 04:48:16.087034 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:16.087004    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q5bzq_530be5d0-8d48-499a-976d-0df89dec736e/limitador/0.log"
Jul 30 04:48:16.180982 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:16.180949    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-psgzj_321a2104-8ee0-4d45-86af-a89aa3e9be2d/manager/0.log"
Jul 30 04:48:17.898756 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:17.898699    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/node-exporter/0.log"
Jul 30 04:48:17.925738 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:17.925712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/kube-rbac-proxy/0.log"
Jul 30 04:48:17.946743 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:17.946656    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99blf_d010c36a-0e87-479f-b4ee-ab12a3ecf6f9/init-textfile/0.log"
Jul 30 04:48:21.033615 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.033580    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"]
Jul 30 04:48:21.041250 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.041223    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.048809 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.048777    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"]
Jul 30 04:48:21.084042 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.083998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-podres\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.084232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.084079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ff7j\" (UniqueName: \"kubernetes.io/projected/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-kube-api-access-5ff7j\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.084232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.084108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-sys\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.084232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.084140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-proc\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.084232 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.084209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-lib-modules\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185462 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-lib-modules\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-podres\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ff7j\" (UniqueName: \"kubernetes.io/projected/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-kube-api-access-5ff7j\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-sys\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185681 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-proc\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185935 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-proc\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.185935 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-lib-modules\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.186047 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.185960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-podres\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.186221 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.186195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-sys\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.196124 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.196098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ff7j\" (UniqueName: \"kubernetes.io/projected/abf1a3c9-e843-4fa3-8c3e-32a9589e8284-kube-api-access-5ff7j\") pod \"perf-node-gather-daemonset-fr9c5\" (UID: \"abf1a3c9-e843-4fa3-8c3e-32a9589e8284\") " pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.354954 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.354856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:21.519619 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.519423    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"]
Jul 30 04:48:21.521965 ip-10-0-138-44 kubenswrapper[2562]: W0730 04:48:21.521935    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podabf1a3c9_e843_4fa3_8c3e_32a9589e8284.slice/crio-9ef0501b4125b445e7dbd4aed2b30f37253fd5194fa057bd9db007cbcbb0fa4e WatchSource:0}: Error finding container 9ef0501b4125b445e7dbd4aed2b30f37253fd5194fa057bd9db007cbcbb0fa4e: Status 404 returned error can't find the container with id 9ef0501b4125b445e7dbd4aed2b30f37253fd5194fa057bd9db007cbcbb0fa4e
Jul 30 04:48:21.953474 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:21.953443    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-d9jfp_c66f94e4-e6c8-4c5c-9e2a-14b2f4b2d5db/volume-data-source-validator/0.log"
Jul 30 04:48:22.005615 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:22.004937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5" event={"ID":"abf1a3c9-e843-4fa3-8c3e-32a9589e8284","Type":"ContainerStarted","Data":"ddd07326fff27289d584cce08e6a5bf3d21ba8015e184ccd4e62552e97610b5e"}
Jul 30 04:48:22.005615 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:22.005001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5" event={"ID":"abf1a3c9-e843-4fa3-8c3e-32a9589e8284","Type":"ContainerStarted","Data":"9ef0501b4125b445e7dbd4aed2b30f37253fd5194fa057bd9db007cbcbb0fa4e"}
Jul 30 04:48:22.005845 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:22.005637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:22.024725 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:22.024670    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5" podStartSLOduration=1.024650855 podStartE2EDuration="1.024650855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 04:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 04:48:22.022598157 +0000 UTC m=+3794.839534634" watchObservedRunningTime="2026-07-30 04:48:22.024650855 +0000 UTC m=+3794.841587332"
Jul 30 04:48:23.095942 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:23.095905    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvbds_ac597b2b-d087-4705-850e-c741a9a3c63e/dns/0.log"
Jul 30 04:48:23.124009 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:23.123987    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvbds_ac597b2b-d087-4705-850e-c741a9a3c63e/kube-rbac-proxy/0.log"
Jul 30 04:48:23.366849 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:23.366773    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrzfr_35991ee7-db6e-4812-94fc-3d81199178b6/dns-node-resolver/0.log"
Jul 30 04:48:23.923428 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:23.923398    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f7b4dff4f-ddcfr_acf349c0-f872-405e-b222-a80397d52f0f/registry/0.log"
Jul 30 04:48:23.974568 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:23.974542    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cbzzj_d28a9969-37d4-4242-a701-75478e599497/node-ca/0.log"
Jul 30 04:48:25.092826 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:25.092796    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58ccc58984-n82rr_10fd0823-093e-4bfc-97ac-a14dad793f8c/kube-auth-proxy/0.log"
Jul 30 04:48:25.231695 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:25.231658    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b9dc4d85-dz8bz_d1347232-d5a8-4c77-b0de-2609a00e9b05/router/0.log"
Jul 30 04:48:26.019989 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:26.019959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6fhsw_76f692da-ea9a-4623-8ee6-20a20440f4d7/serve-healthcheck-canary/0.log"
Jul 30 04:48:26.641157 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:26.641129    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2jpw_0e839a7c-89b6-4915-a518-16ecbd17e771/kube-rbac-proxy/0.log"
Jul 30 04:48:26.663405 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:26.663366    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2jpw_0e839a7c-89b6-4915-a518-16ecbd17e771/exporter/0.log"
Jul 30 04:48:26.687161 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:26.687125    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v2jpw_0e839a7c-89b6-4915-a518-16ecbd17e771/extractor/0.log"
Jul 30 04:48:28.698236 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:28.698211    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bdxxf_03376731-166b-441e-91c4-c5d2e3152555/manager/0.log"
Jul 30 04:48:28.997281 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:28.997233    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-52dzh_a793978b-1b44-4de8-950c-9a2d759a5a06/manager/0.log"
Jul 30 04:48:29.029337 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:29.029305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qdzjx/perf-node-gather-daemonset-fr9c5"
Jul 30 04:48:29.173196 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:29.173168    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2hlrv_490420af-8cae-4a85-86e2-596af1e63beb/server/0.log"
Jul 30 04:48:29.266552 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:29.266414    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jc2b7_81fc883b-e8d5-4b2b-b791-5a5d6daf066f/manager/0.log"
Jul 30 04:48:29.420412 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:29.420384    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-n9nv5_e1031033-e23b-4455-a957-7e449137dfc2/manager/0.log"
Jul 30 04:48:29.442011 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:29.441987    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-599789d45f-n9nv5_e1031033-e23b-4455-a957-7e449137dfc2/copy-manifests/0.log"
Jul 30 04:48:30.768428 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:30.768391    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58ccc58984-n82rr_10fd0823-093e-4bfc-97ac-a14dad793f8c/kube-auth-proxy/0.log"
Jul 30 04:48:30.877964 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:30.877937    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b9dc4d85-dz8bz_d1347232-d5a8-4c77-b0de-2609a00e9b05/router/0.log"
Jul 30 04:48:32.354845 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:32.354780    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58ccc58984-n82rr_10fd0823-093e-4bfc-97ac-a14dad793f8c/kube-auth-proxy/0.log"
Jul 30 04:48:32.461968 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:32.461942    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77b9dc4d85-dz8bz_d1347232-d5a8-4c77-b0de-2609a00e9b05/router/0.log"
Jul 30 04:48:36.108388 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:36.108353    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t_55773a70-3b09-4801-82a8-e1dee6bdf633/kube-storage-version-migrator-operator/1.log"
Jul 30 04:48:36.109516 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:36.109495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-k8k4t_55773a70-3b09-4801-82a8-e1dee6bdf633/kube-storage-version-migrator-operator/0.log"
Jul 30 04:48:37.074492 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.074463    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6nnwv_49a91781-5cb3-40c3-acba-91c9b1e807c4/kube-multus/0.log"
Jul 30 04:48:37.256866 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.256840    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/kube-multus-additional-cni-plugins/0.log"
Jul 30 04:48:37.282890 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.282865    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/egress-router-binary-copy/0.log"
Jul 30 04:48:37.305827 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.305792    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/cni-plugins/0.log"
Jul 30 04:48:37.326778 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.326722    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/bond-cni-plugin/0.log"
Jul 30 04:48:37.346808 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.346784    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/routeoverride-cni/0.log"
Jul 30 04:48:37.367294 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.367277    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/whereabouts-cni-bincopy/0.log"
Jul 30 04:48:37.390258 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.390234    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lgjs4_efb3b2a5-0f80-4320-b493-a18d2917a8f3/whereabouts-cni/0.log"
Jul 30 04:48:37.732877 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.732852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svjms_d6717a3e-b8b5-4bc5-bd06-bf26a1f41837/network-metrics-daemon/0.log"
Jul 30 04:48:37.751010 ip-10-0-138-44 kubenswrapper[2562]: I0730 04:48:37.750988    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svjms_d6717a3e-b8b5-4bc5-bd06-bf26a1f41837/kube-rbac-proxy/0.log"