Jul 23 00:07:45.011281 ip-10-0-141-133 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.465843    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471283    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471297    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471303    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471308    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471311    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471314    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471319    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471322    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.494240 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471324    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471327    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471332    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471334    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471337    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471341    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471344    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471347    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471357    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471361    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471364    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471367    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471370    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471373    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471376    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471379    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471381    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471386    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471389    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471391    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.506698 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471396    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471400    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471402    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471407    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471409    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471412    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471415    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471420    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471422    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471425    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471428    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471432    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471435    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471442    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471447    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471450    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471455    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471459    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471462    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471466    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.509193 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471469    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471472    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471477    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471480    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471483    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471488    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471492    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471495    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471500    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471502    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471505    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471509    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471512    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471515    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471518    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471522    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471525    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471527    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471533    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.510921 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471536    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471539    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471545    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471549    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471551    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471556    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471558    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471561    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471565    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471569    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471572    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471575    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471578    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471581    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471583    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471587    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471590    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471592    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471599    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.513113 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471602    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471605    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471608    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471613    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471615    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471617    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471623    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471628    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471632    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471637    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471641    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471645    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.471649    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472441    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472448    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472451    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472454    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472457    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472461    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472463    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.515074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472466    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472469    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472474    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472479    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472482    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472485    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472488    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472491    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472494    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472497    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472500    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472503    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472506    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472509    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472511    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472514    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472517    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472520    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472523    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472525    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.517830 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472528    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472530    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472533    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472536    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472539    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472542    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472544    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472547    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472549    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472552    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472554    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472557    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472560    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472563    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472565    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472569    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472571    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472574    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472577    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472580    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.520026 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472583    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472585    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472588    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472590    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472593    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472597    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472600    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472603    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472606    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472611    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472613    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472616    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472618    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472621    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472624    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472627    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472629    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472632    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472634    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472637    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.522230 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472639    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472642    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472644    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472647    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472651    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472655    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472657    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472660    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472662    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472664    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472667    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472670    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472672    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472674    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472677    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472679    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472682    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472684    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472687    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.523903 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472689    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472692    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472696    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472699    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472701    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472706    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472709    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472712    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472716    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472718    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472721    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472723    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.472727    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473098    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473104    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473107    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473110    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473116    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473119    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.526120 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473122    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473124    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473127    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473130    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473133    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473135    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473138    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473141    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473143    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473146    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473149    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473151    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473154    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473156    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473161    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473164    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473166    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473170    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473173    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473176    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.527784 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473179    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473181    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473184    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473186    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473189    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473191    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473194    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473197    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473199    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473201    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473204    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473207    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473210    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473213    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473215    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473218    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473220    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473223    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473226    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473228    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.529853 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473231    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473234    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473236    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473239    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473241    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473244    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473248    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473252    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473255    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473258    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473260    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473263    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473266    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473269    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473271    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473274    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473276    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473279    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473281    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.531621 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473284    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473286    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473289    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473292    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473294    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473297    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473300    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473302    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473305    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473309    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473312    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473315    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473318    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473321    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473323    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473326    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473329    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473331    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473334    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473338    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.533850 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473340    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473343    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473346    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473349    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473352    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473354    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473357    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473359    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473362    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473365    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473367    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473371    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.473374    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473444    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473451    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473457    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473461    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473466    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473469    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473474    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 00:07:45.535888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473478    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473481    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473484    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473488    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473491    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473494    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473497    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473500    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473503    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473505    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473508    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473514    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473517    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473520    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473523    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473527    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473530    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473533    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473536    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473543    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473548    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473551    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473554    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473557    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 00:07:45.538422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473560    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473564    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473567    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473570    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473573    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473576    2565 flags.go:64] FLAG: --enable-server="true"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473579    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473584    2565 flags.go:64] FLAG: --event-burst="100"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473587    2565 flags.go:64] FLAG: --event-qps="50"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473590    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473593    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473596    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473599    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473602    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473605    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473608    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473611    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473614    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473617    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473620    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473623    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473626    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473639    2565 flags.go:64] FLAG: --feature-gates=""
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473643    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473647    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 00:07:45.540361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473650    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473653    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473656    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473661    2565 flags.go:64] FLAG: --help="false"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473664    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473667    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473670    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473673    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473676    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473680    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473682    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473685    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473688    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473691    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473694    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473698    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473701    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473703    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473706    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473709    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473712    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473715    2565 flags.go:64] FLAG: --lock-file=""
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473718    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473720    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 00:07:45.542843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473723    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473728    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473733    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473735    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473738    2565 flags.go:64] FLAG: --logging-format="text"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473741    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473744    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473747    2565 flags.go:64] FLAG: --manifest-url=""
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473750    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473754    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473757    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473762    2565 flags.go:64] FLAG: --max-pods="110"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473765    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473768    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473771    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473774    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473777    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473780    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473783    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473789    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473792    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473795    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473798    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 23 00:07:45.584539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473801    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 00:07:45.549637 ip-10-0-141-133 systemd[1]: Started Kubernetes Kubelet.
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473807    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473809    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473812    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473815    2565 flags.go:64] FLAG: --port="10250"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473818    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473821    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088e31e7dd0f10447"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473824    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473827    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473830    2565 flags.go:64] FLAG: --register-node="true"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473833    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473836    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473840    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473843    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473846    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473854    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473857    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473860    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473863    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473866    2565 flags.go:64] FLAG: --runonce="false"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473870    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473873    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473876    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473879    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473882    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473884    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 00:07:45.614111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473888    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473891    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473894    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473897    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473900    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473903    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473906    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473909    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473912    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473917    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473920    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473923    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473927    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473929    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473932    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473935    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473938    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473940    2565 flags.go:64] FLAG: --v="2"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473945    2565 flags.go:64] FLAG: --version="false"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473949    2565 flags.go:64] FLAG: --vmodule=""
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473952    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.473955    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474090    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474094    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.846769 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474097    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474100    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474104    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474107    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474110    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474113    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474116    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474119    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474121    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474124    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474126    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474129    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474131    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474134    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474137    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474139    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474145    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474148    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474152    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474156    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.865037 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474158    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474161    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474164    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474166    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474169    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474171    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474175    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474178    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474181    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474183    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474186    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474188    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474191    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474194    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474198    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474201    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474203    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474206    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474209    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474211    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.866016 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474235    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474239    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474242    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474245    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474248    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474251    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474254    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474256    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474259    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474262    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474265    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474267    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474270    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474272    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474276    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474279    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474283    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474286    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474290    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474293    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.866697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474295    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474298    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474301    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474303    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474306    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474308    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474312    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474315    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474318    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474320    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474323    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474325    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474328    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474330    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474333    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474335    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474338    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474341    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474343    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.867336 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474346    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474349    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474351    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474360    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474362    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474365    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474369    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474372    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474375    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474378    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474383    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474386    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474389    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474392    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474395    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474397    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.868002 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.474400    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.475172    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.482276    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.482292    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.482310    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.482317    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482374    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482378    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482382    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482385    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482388    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482392    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482394    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482397    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482400    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.868545 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482402    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482404    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482407    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482411    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482414    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482416    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482419    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482425    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482429    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482431    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482434    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482437    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482439    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482442    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482446    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482450    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482453    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482455    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482458    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482461    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.869077 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482464    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482467    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482469    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482472    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482474    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482477    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482479    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482482    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482484    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482487    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482490    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482492    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482495    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482497    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482501    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482505    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482508    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482511    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482514    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.869689 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482520    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482523    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482526    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482529    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482532    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482534    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482537    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482540    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482542    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482545    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482548    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482550    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482553    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482555    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482559    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482561    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482564    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482566    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482569    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.870375 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482572    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482574    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482577    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482579    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482582    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482584    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482587    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482589    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482592    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482596    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482598    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482601    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482604    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482609    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482612    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482614    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482617    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482620    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482623    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482625    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.870948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482627    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482630    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482633    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482635    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482638    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482641    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482643    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482646    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482649    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482651    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482654    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482657    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.482662    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482783    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482787    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 00:07:45.871597 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482790    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482793    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482796    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482798    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482801    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482803    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482807    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482810    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482812    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482815    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482817    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482821    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482823    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482826    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482829    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482831    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482834    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482836    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482839    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482841    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 00:07:45.872101 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482844    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482847    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482849    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482852    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482854    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482857    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482860    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482863    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482865    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482869    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482873    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482876    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482879    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482881    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482884    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482887    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482889    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482892    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482896    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 00:07:45.872725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482898    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482900    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482903    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482906    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482910    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482912    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482916    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482919    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482922    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482925    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482927    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482930    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482932    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482935    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482938    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482940    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482943    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482945    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482948    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482950    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 00:07:45.873257 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482953    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482956    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482959    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482961    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482964    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482981    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482984    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482987    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482989    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482992    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482995    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.482999    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483002    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483005    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483007    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483010    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483014    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483016    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483019    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 00:07:45.873948 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483022    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483026    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483029    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483032    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483035    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483038    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483040    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483043    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483045    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483048    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483050    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483053    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483055    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483058    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483061    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483063    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483066    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483068    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 00:07:45.874845 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:45.483071    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.483075    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.483173    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.485138    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.486881    2565 server.go:1019] "Starting client certificate rotation"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.486986    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.487022    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.489551    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.493371    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.506229    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.508407    2565 log.go:25] "Validated CRI v1 image API"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.509283    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 00:07:45.875747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.513767    2565 fs.go:135] Filesystem UUIDs: map[31b4afdc-0a4e-4452-ad17-4e21f6567d11:/dev/nvme0n1p3 45686652-35d4-455d-ad72-b921b7202a9e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 23 00:07:45.876423 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.513788    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 00:07:45.876423 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.515425    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.519887    2565 manager.go:216] Machine: {Timestamp:2026-07-23 00:07:45.518128518 +0000 UTC m=+0.371698184 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097010 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214ff8b321253ab84184420b38159e SystemUUID:ec214ff8-b321-253a-b841-84420b38159e BootID:85012abe-8214-405e-910a-f0e6028c4b69 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:90:98:c8:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:90:98:c8:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:e7:50:7a:8f:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.520041    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.520150    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.522207    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.522235    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-133.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.523887    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.523898    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.523910    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.527776    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.530365    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533605    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533730    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533761    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533772    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533787    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533796    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.533805    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 00:07:45.878999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.534211    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.534233    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.537524    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.539958    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.539999    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541137    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541150    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541156    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541161    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541170    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541175    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541183    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541189    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541195    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541201    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541209    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541214    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541759    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.541769    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.544553    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-133.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.546319    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.546326    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.548663    2565 server.go:1268] "Started kubelet"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.548731    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.548821    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.548881    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.553469    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.555022    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.555349    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.558472    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-133.ec2.internal.18c4c28d8c7b5f5e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-133.ec2.internal,UID:ip-10-0-141-133.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-133.ec2.internal,},FirstTimestamp:2026-07-23 00:07:45.548631902 +0000 UTC m=+0.402201567,LastTimestamp:2026-07-23 00:07:45.548631902 +0000 UTC m=+0.402201567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-133.ec2.internal,}"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560127    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560133    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.560272    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560704    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560732    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560754    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560902    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560912    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560947    2565 factory.go:55] Registering systemd factory
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.560959    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561178    2565 factory.go:153] Registering CRI-O factory
Jul 23 00:07:45.880266 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561195    2565 factory.go:223] Registration of the crio container factory successfully
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561249    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561275    2565 factory.go:103] Registering Raw factory
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561289    2565 manager.go:1196] Started watching for new ooms in manager
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.561786    2565 manager.go:349] Starting recovery of all containers
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.564776    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9ptx"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.570861    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9ptx"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.571842    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.571853    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-133.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.572706    2565 manager.go:354] Recovery completed
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.576720    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581073    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581095    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581108    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581743    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581754    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581764    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.581770    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.582907    2565 policy_none.go:49] "None policy: Start"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.582919    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.582928    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.583892    2565 policy_none.go:47] "Start"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.584521    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.584551    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.584562    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.660614    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.689699    2565 manager.go:337] "Starting Device Plugin manager"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.689747    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.689760    2565 server.go:87] "Starting device plugin registration server"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.690083    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.690096    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.690176    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.690264    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.690273    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.690805    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.690861    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.717996    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.719203    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.719240    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.719267    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.719310    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.721887    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.777756    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-133.ec2.internal\" not found" node="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.790861    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.791846    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.882402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.791874    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.791885    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.791909    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.801846    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.801866    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-133.ec2.internal\": node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.819640    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"]
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.819712    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.821185    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.821211    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.821221    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.822386    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.822499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.822533    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823082    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823101    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823111    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823139    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823156    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.823165    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.824297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.824327    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.825037    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.825058    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.825070    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeHasSufficientPID"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.829177    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.861863    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-133.ec2.internal\" not found" node="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.862561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.862585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.862622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/642dc77bb1e95d4ad1afe2c58591ff7f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-133.ec2.internal\" (UID: \"642dc77bb1e95d4ad1afe2c58591ff7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:45.884346 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.866418    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-133.ec2.internal\" not found" node="ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:45.930229    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/642dc77bb1e95d4ad1afe2c58591ff7f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-133.ec2.internal\" (UID: \"642dc77bb1e95d4ad1afe2c58591ff7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/642dc77bb1e95d4ad1afe2c58591ff7f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-133.ec2.internal\" (UID: \"642dc77bb1e95d4ad1afe2c58591ff7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:45.963893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed36492455dccc8c0cba69df6878eb97-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal\" (UID: \"ed36492455dccc8c0cba69df6878eb97\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.081254 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.030421    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.255494 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.131015    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.255494 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.166050    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.255494 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.169116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.255494 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.231458    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.471457 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.331990    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.471457 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.432176    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.487760    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.487908    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.534135    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.575449    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:02:46 +0000 UTC" deadline="2028-02-08 04:45:37.880490788 +0000 UTC"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.575473    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13564h37m51.305020759s"
Jul 23 00:07:46.670965 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:46.635162    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-133.ec2.internal\" not found"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.691855    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.718354    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:46.743668    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642dc77bb1e95d4ad1afe2c58591ff7f.slice/crio-ea12782c13320facd8823927e331e3e65117209b914289ff6e396244860153f9 WatchSource:0}: Error finding container ea12782c13320facd8823927e331e3e65117209b914289ff6e396244860153f9: Status 404 returned error can't find the container with id ea12782c13320facd8823927e331e3e65117209b914289ff6e396244860153f9
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:46.744391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded36492455dccc8c0cba69df6878eb97.slice/crio-f65ead1a83999066aeaed5c292414e016fbcbb734434c9169d611d033ebb882d WatchSource:0}: Error finding container f65ead1a83999066aeaed5c292414e016fbcbb734434c9169d611d033ebb882d: Status 404 returned error can't find the container with id f65ead1a83999066aeaed5c292414e016fbcbb734434c9169d611d033ebb882d
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.751003    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.761092    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.774116    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.776256    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal"
Jul 23 00:07:46.904532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:46.786080    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 00:07:47.129306 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.021102    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.535205    2565 apiserver.go:52] "Watching apiserver"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.549906    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.552959    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-v5btk","openshift-dns/node-resolver-dwthh","openshift-multus/multus-additional-cni-plugins-fxk7c","openshift-multus/multus-bj99z","openshift-network-operator/iptables-alerter-qsz4z","openshift-ovn-kubernetes/ovnkube-node-ghsx9","kube-system/global-pull-secret-syncer-j9hf9","kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv","openshift-cluster-node-tuning-operator/tuned-sgrtb","openshift-image-registry/node-ca-7p5th","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal","openshift-multus/network-metrics-daemon-8vgc5","openshift-network-diagnostics/network-check-target-khbk4"]
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.554980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.557793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zmzbv\""
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.559604    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.560363    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-run\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-etc-tuned\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-conf\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-sys\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-lib-modules\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-var-lib-kubelet\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysconfig\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-host\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.584008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.583973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-modprobe-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.597648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.597023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-kubernetes\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.597648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.597088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.597648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.597122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-systemd\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.597648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.597544    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.597648 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.597634    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:47.600285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.598546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.600285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.598727    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 00:07:47.600285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.598944    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2hbx"
Jul 23 00:07:47.600285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.599375    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.601136 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.601119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.602235 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.601598    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n2hbx"
Jul 23 00:07:47.602235 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.602227    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 00:07:47.602740 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.602404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.602740 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.602484    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bccq7\""
Jul 23 00:07:47.602740 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.602722    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.603201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.602990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 00:07:47.603201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.603058    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 00:07:47.603201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.603186    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.603861 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.603832    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 00:07:47.604213 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.604193    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-txrt7\""
Jul 23 00:07:47.604859 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.604436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.606109 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.605743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kw644\""
Jul 23 00:07:47.606109 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.605754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.606109 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.605983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.606908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.606488    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 00:07:47.606908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.606559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.607355 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.607264    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.608035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.607745    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 00:07:47.608035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.608022    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 00:07:47.608624 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.608298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:47.608624 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.608368    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:47.609543 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbgfl\""
Jul 23 00:07:47.609543 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609530    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 00:07:47.610520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609625    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.610520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609698    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 00:07:47.610520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9gf4\""
Jul 23 00:07:47.610520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609718    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 00:07:47.610520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.609921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.611513 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.611490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.613740 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.613720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.615359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.615334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.615747 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.615398    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:47.628821 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.628800    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 00:07:47.629731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.629670    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.629925    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.629927    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.630038    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltqsf\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.629929    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.630261    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.630598    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 00:07:47.630831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.630683    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x5smg\""
Jul 23 00:07:47.631324 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.631301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mpw27\""
Jul 23 00:07:47.631435 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.631335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.631697 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.631677    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 00:07:47.661782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.661760    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 00:07:47.697602 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-systemd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.697602 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2th\" (UniqueName: \"kubernetes.io/projected/d0779a56-6596-4699-9aca-fa8d312ebda9-kube-api-access-8s2th\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.697602 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mn9\" (UniqueName: \"kubernetes.io/projected/d283f102-2e7d-4d9f-a493-aed1369cd78d-kube-api-access-s6mn9\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.697602 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-conf\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.697602 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-sys\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-lib-modules\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-var-lib-kubelet\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8krf\" (UniqueName: \"kubernetes.io/projected/3c736f2d-3dc5-4f07-963c-c5deb04f105f-kube-api-access-t8krf\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-sys\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-slash\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-conf\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-config\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-lib-modules\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jd5s\" (UniqueName: \"kubernetes.io/projected/15208ffd-1eda-488b-9099-d5133d306d2e-kube-api-access-8jd5s\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-var-lib-kubelet\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtfb\" (UniqueName: \"kubernetes.io/projected/3ac5aa4f-7a25-4e1c-9f7a-579072177156-kube-api-access-jmtfb\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-multus-certs\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-systemd-units\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-netd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.698360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-modprobe-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-conf-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.697990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-multus-daemon-config\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-etc-kubernetes\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d283f102-2e7d-4d9f-a493-aed1369cd78d-host\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-os-release\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4c9\" (UniqueName: \"kubernetes.io/projected/e5bc1acf-579e-4827-bbc5-3877510d9f03-kube-api-access-4t4c9\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-registration-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-modprobe-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-device-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysconfig\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-bin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-kubelet\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysconfig\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-sys-fs\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-k8s-cni-cncf-io\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.699192 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-netns\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-etc-tuned\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cnibin\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-cnibin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-hostroot\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629cb05f-3479-4f5b-a410-1c32ec4d52d1-tmp-dir\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-etc-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-log-socket\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-tmp\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdsz\" (UniqueName: \"kubernetes.io/projected/a609c562-984b-4e40-a05d-58f2227174de-kube-api-access-hgdsz\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6tj\" (UniqueName: \"kubernetes.io/projected/629cb05f-3479-4f5b-a410-1c32ec4d52d1-kube-api-access-6n6tj\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698681    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c68d68-0526-45e1-966a-883bb02511be-agent-certs\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-env-overrides\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d283f102-2e7d-4d9f-a493-aed1369cd78d-serviceca\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-system-cni-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-ovn\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0779a56-6596-4699-9aca-fa8d312ebda9-ovn-node-metrics-cert\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.698996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2579d\" (UniqueName: \"kubernetes.io/projected/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kube-api-access-2579d\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-system-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-os-release\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-socket-dir-parent\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-sysctl-d\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-multus\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5bc1acf-579e-4827-bbc5-3877510d9f03-host-slash\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-netns\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-var-lib-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.700921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-cni-binary-copy\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/629cb05f-3479-4f5b-a410-1c32ec4d52d1-hosts-file\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c68d68-0526-45e1-966a-883bb02511be-konnectivity-ca\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-node-log\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-socket-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-host\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-host\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5bc1acf-579e-4827-bbc5-3877510d9f03-iptables-alerter-script\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-kubelet\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-bin\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-script-lib\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-dbus\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.701756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.699928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-kubernetes\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-kubelet-config\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-systemd\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-run\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-run\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-kubernetes\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.702590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.700466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15208ffd-1eda-488b-9099-d5133d306d2e-etc-systemd\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.706977 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.706917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-etc-tuned\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.726164 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.726107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal" event={"ID":"ed36492455dccc8c0cba69df6878eb97","Type":"ContainerStarted","Data":"f65ead1a83999066aeaed5c292414e016fbcbb734434c9169d611d033ebb882d"}
Jul 23 00:07:47.728100 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.728071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal" event={"ID":"642dc77bb1e95d4ad1afe2c58591ff7f","Type":"ContainerStarted","Data":"ea12782c13320facd8823927e331e3e65117209b914289ff6e396244860153f9"}
Jul 23 00:07:47.764579 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.764551    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 00:07:47.801203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-ovn\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0779a56-6596-4699-9aca-fa8d312ebda9-ovn-node-metrics-cert\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2579d\" (UniqueName: \"kubernetes.io/projected/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kube-api-access-2579d\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-system-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-os-release\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.801386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-socket-dir-parent\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-multus\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5bc1acf-579e-4827-bbc5-3877510d9f03-host-slash\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-netns\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-var-lib-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-cni-binary-copy\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/629cb05f-3479-4f5b-a410-1c32ec4d52d1-hosts-file\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c68d68-0526-45e1-966a-883bb02511be-konnectivity-ca\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-node-log\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-socket-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5bc1acf-579e-4827-bbc5-3877510d9f03-iptables-alerter-script\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-kubelet\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-bin\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-script-lib\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.801791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-dbus\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-var-lib-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-kubelet-config\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-systemd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-kubelet-config\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.802753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.801913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2th\" (UniqueName: \"kubernetes.io/projected/d0779a56-6596-4699-9aca-fa8d312ebda9-kube-api-access-8s2th\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.803182 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-cni-binary-copy\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.803365 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.803459 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-socket-dir-parent\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.803538 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-ovn\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.803538 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.803837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-etc-selinux\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.804024    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/629cb05f-3479-4f5b-a410-1c32ec4d52d1-hosts-file\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.804667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c68d68-0526-45e1-966a-883bb02511be-konnectivity-ca\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.804448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5bc1acf-579e-4827-bbc5-3877510d9f03-iptables-alerter-script\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.805705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-kubelet\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.805769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-bin\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.806322    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.806447    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:48.306388653 +0000 UTC m=+3.159958327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-script-lib\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-node-log\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-dbus\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-socket-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.809071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.806937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-system-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-run-netns\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-multus\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-os-release\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5bc1acf-579e-4827-bbc5-3877510d9f03-host-slash\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-run-systemd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mn9\" (UniqueName: \"kubernetes.io/projected/d283f102-2e7d-4d9f-a493-aed1369cd78d-kube-api-access-s6mn9\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8krf\" (UniqueName: \"kubernetes.io/projected/3c736f2d-3dc5-4f07-963c-c5deb04f105f-kube-api-access-t8krf\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-slash\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-config\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jd5s\" (UniqueName: \"kubernetes.io/projected/15208ffd-1eda-488b-9099-d5133d306d2e-kube-api-access-8jd5s\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtfb\" (UniqueName: \"kubernetes.io/projected/3ac5aa4f-7a25-4e1c-9f7a-579072177156-kube-api-access-jmtfb\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.807852    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.807948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:48.307904739 +0000 UTC m=+3.161474407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.807944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-slash\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-multus-certs\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.809959 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-systemd-units\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-netd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-conf-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-multus-daemon-config\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-etc-kubernetes\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d283f102-2e7d-4d9f-a493-aed1369cd78d-host\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-os-release\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4c9\" (UniqueName: \"kubernetes.io/projected/e5bc1acf-579e-4827-bbc5-3877510d9f03-kube-api-access-4t4c9\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-ovnkube-config\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-registration-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-systemd-units\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-device-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-registration-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-bin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-kubelet\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.810831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-host-cni-netd\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-sys-fs\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-conf-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-k8s-cni-cncf-io\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-sys-fs\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-netns\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-cni-bin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cnibin\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-cnibin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-device-dir\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-var-lib-kubelet\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-os-release\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d283f102-2e7d-4d9f-a493-aed1369cd78d-host\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a609c562-984b-4e40-a05d-58f2227174de-multus-daemon-config\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-k8s-cni-cncf-io\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-multus-cni-dir\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-etc-kubernetes\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.811666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-netns\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-cnibin\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.809688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-cnibin\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.808962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-host-run-multus-certs\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-hostroot\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a609c562-984b-4e40-a05d-58f2227174de-hostroot\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629cb05f-3479-4f5b-a410-1c32ec4d52d1-tmp-dir\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-etc-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-log-socket\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-tmp\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdsz\" (UniqueName: \"kubernetes.io/projected/a609c562-984b-4e40-a05d-58f2227174de-kube-api-access-hgdsz\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6tj\" (UniqueName: \"kubernetes.io/projected/629cb05f-3479-4f5b-a410-1c32ec4d52d1-kube-api-access-6n6tj\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c68d68-0526-45e1-966a-883bb02511be-agent-certs\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-env-overrides\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d283f102-2e7d-4d9f-a493-aed1369cd78d-serviceca\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629cb05f-3479-4f5b-a410-1c32ec4d52d1-tmp-dir\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.812605 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-system-cni-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.813573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.811717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-log-socket\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.813573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.812613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c736f2d-3dc5-4f07-963c-c5deb04f105f-system-cni-dir\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.813573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.813194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c736f2d-3dc5-4f07-963c-c5deb04f105f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.813573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.813235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d283f102-2e7d-4d9f-a493-aed1369cd78d-serviceca\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.813573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.812260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0779a56-6596-4699-9aca-fa8d312ebda9-etc-openvswitch\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.813772 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.813601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0779a56-6596-4699-9aca-fa8d312ebda9-env-overrides\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.814418 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.814393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15208ffd-1eda-488b-9099-d5133d306d2e-tmp\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.816139 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.816111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0779a56-6596-4699-9aca-fa8d312ebda9-ovn-node-metrics-cert\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.816259 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.816119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c68d68-0526-45e1-966a-883bb02511be-agent-certs\") pod \"konnectivity-agent-v5btk\" (UID: \"b6c68d68-0526-45e1-966a-883bb02511be\") " pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.816259 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.816193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2th\" (UniqueName: \"kubernetes.io/projected/d0779a56-6596-4699-9aca-fa8d312ebda9-kube-api-access-8s2th\") pod \"ovnkube-node-ghsx9\" (UID: \"d0779a56-6596-4699-9aca-fa8d312ebda9\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.819075 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.818912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mn9\" (UniqueName: \"kubernetes.io/projected/d283f102-2e7d-4d9f-a493-aed1369cd78d-kube-api-access-s6mn9\") pod \"node-ca-7p5th\" (UID: \"d283f102-2e7d-4d9f-a493-aed1369cd78d\") " pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:47.819697 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.819235    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:07:47.819697 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.819266    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:07:47.819697 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.819283    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:47.819697 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:47.819377    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:48.319358749 +0000 UTC m=+3.172928402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:47.824254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.822858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jd5s\" (UniqueName: \"kubernetes.io/projected/15208ffd-1eda-488b-9099-d5133d306d2e-kube-api-access-8jd5s\") pod \"tuned-sgrtb\" (UID: \"15208ffd-1eda-488b-9099-d5133d306d2e\") " pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.824254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.823204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6tj\" (UniqueName: \"kubernetes.io/projected/629cb05f-3479-4f5b-a410-1c32ec4d52d1-kube-api-access-6n6tj\") pod \"node-resolver-dwthh\" (UID: \"629cb05f-3479-4f5b-a410-1c32ec4d52d1\") " pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.824644 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.824621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdsz\" (UniqueName: \"kubernetes.io/projected/a609c562-984b-4e40-a05d-58f2227174de-kube-api-access-hgdsz\") pod \"multus-bj99z\" (UID: \"a609c562-984b-4e40-a05d-58f2227174de\") " pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.825800 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.825765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8krf\" (UniqueName: \"kubernetes.io/projected/3c736f2d-3dc5-4f07-963c-c5deb04f105f-kube-api-access-t8krf\") pod \"multus-additional-cni-plugins-fxk7c\" (UID: \"3c736f2d-3dc5-4f07-963c-c5deb04f105f\") " pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.826434 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.826410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4c9\" (UniqueName: \"kubernetes.io/projected/e5bc1acf-579e-4827-bbc5-3877510d9f03-kube-api-access-4t4c9\") pod \"iptables-alerter-qsz4z\" (UID: \"e5bc1acf-579e-4827-bbc5-3877510d9f03\") " pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.826817 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.826797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2579d\" (UniqueName: \"kubernetes.io/projected/d2c52e54-4cd4-45d8-86ad-f2dc73069e38-kube-api-access-2579d\") pod \"aws-ebs-csi-driver-node-lf4tv\" (UID: \"d2c52e54-4cd4-45d8-86ad-f2dc73069e38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.828541 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.828185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtfb\" (UniqueName: \"kubernetes.io/projected/3ac5aa4f-7a25-4e1c-9f7a-579072177156-kube-api-access-jmtfb\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:47.898714 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.898680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgrtb"
Jul 23 00:07:47.912063 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.912032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fxk7c"
Jul 23 00:07:47.921451 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.921427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bj99z"
Jul 23 00:07:47.931817 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.931793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsz4z"
Jul 23 00:07:47.940193 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.940172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:07:47.953806 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.953748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv"
Jul 23 00:07:47.964191 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.964173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwthh"
Jul 23 00:07:47.972698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.972678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:07:47.980087 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:47.980066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7p5th"
Jul 23 00:07:48.315440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.315354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:48.315609 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.315463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:48.315609 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.315502    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:48.315609 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.315589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:49.315564325 +0000 UTC m=+4.169133988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:48.315758 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.315607    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:48.315758 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.315664    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:49.315645796 +0000 UTC m=+4.169215466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:48.416305 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.416272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:48.416471 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.416454    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:07:48.416530 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.416477    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:07:48.416530 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.416492    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:48.416630 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.416548    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:49.41653135 +0000 UTC m=+4.270101003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:48.605134 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.605033    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:02:48 +0000 UTC" deadline="2028-03-05 03:05:00.960122408 +0000 UTC"
Jul 23 00:07:48.605134 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.605082    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14186h57m12.355042857s"
Jul 23 00:07:48.720296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.720267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:48.720788 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.720410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:48.720788 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:48.720468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:48.720788 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:48.720621    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:48.870072 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.870033    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c736f2d_3dc5_4f07_963c_c5deb04f105f.slice/crio-2a5d01919f85b6f9568556463b78e8967b5410e643551cb0250e86ed1d2dde43 WatchSource:0}: Error finding container 2a5d01919f85b6f9568556463b78e8967b5410e643551cb0250e86ed1d2dde43: Status 404 returned error can't find the container with id 2a5d01919f85b6f9568556463b78e8967b5410e643551cb0250e86ed1d2dde43
Jul 23 00:07:48.871466 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.871427    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629cb05f_3479_4f5b_a410_1c32ec4d52d1.slice/crio-87a50c868d3b4ccf69afdcb3f60d6ed996016646c6c9fa3317c063ebbe3f6523 WatchSource:0}: Error finding container 87a50c868d3b4ccf69afdcb3f60d6ed996016646c6c9fa3317c063ebbe3f6523: Status 404 returned error can't find the container with id 87a50c868d3b4ccf69afdcb3f60d6ed996016646c6c9fa3317c063ebbe3f6523
Jul 23 00:07:48.872840 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.872810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda609c562_984b_4e40_a05d_58f2227174de.slice/crio-93885ea79b64b517ad31ea8fad6cff7a6634d6276476b400499e275cd897e34f WatchSource:0}: Error finding container 93885ea79b64b517ad31ea8fad6cff7a6634d6276476b400499e275cd897e34f: Status 404 returned error can't find the container with id 93885ea79b64b517ad31ea8fad6cff7a6634d6276476b400499e275cd897e34f
Jul 23 00:07:48.876472 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.876421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c52e54_4cd4_45d8_86ad_f2dc73069e38.slice/crio-0f4b3834e2708f7be8086b4c68d0471848e935b14268e9427721f053e896b491 WatchSource:0}: Error finding container 0f4b3834e2708f7be8086b4c68d0471848e935b14268e9427721f053e896b491: Status 404 returned error can't find the container with id 0f4b3834e2708f7be8086b4c68d0471848e935b14268e9427721f053e896b491
Jul 23 00:07:48.877544 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.877408    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bc1acf_579e_4827_bbc5_3877510d9f03.slice/crio-8cacc0c713cb4a05fed02b2bd4c836467eadeced6ed116fef4a36fd2b502fc76 WatchSource:0}: Error finding container 8cacc0c713cb4a05fed02b2bd4c836467eadeced6ed116fef4a36fd2b502fc76: Status 404 returned error can't find the container with id 8cacc0c713cb4a05fed02b2bd4c836467eadeced6ed116fef4a36fd2b502fc76
Jul 23 00:07:48.879697 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.878905    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0779a56_6596_4699_9aca_fa8d312ebda9.slice/crio-18ec1b058d6ff07e1abdf56dbf9195e08a5b1441c1252e362b58e3b9b58ce501 WatchSource:0}: Error finding container 18ec1b058d6ff07e1abdf56dbf9195e08a5b1441c1252e362b58e3b9b58ce501: Status 404 returned error can't find the container with id 18ec1b058d6ff07e1abdf56dbf9195e08a5b1441c1252e362b58e3b9b58ce501
Jul 23 00:07:48.883063 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:07:48.882560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c68d68_0526_45e1_966a_883bb02511be.slice/crio-bfc0f35abf466176171bd4184380ad293ce973b992414d1f8e0510ab4b3e8426 WatchSource:0}: Error finding container bfc0f35abf466176171bd4184380ad293ce973b992414d1f8e0510ab4b3e8426: Status 404 returned error can't find the container with id bfc0f35abf466176171bd4184380ad293ce973b992414d1f8e0510ab4b3e8426
Jul 23 00:07:49.322160 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.322110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:49.322338 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.322236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:49.322338 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.322300    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:49.322459 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.322349    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:49.322459 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.322384    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:51.322363371 +0000 UTC m=+6.175933054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:49.322459 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.322412    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:51.322396987 +0000 UTC m=+6.175966648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:49.423142 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.423005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:49.423359 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.423234    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:07:49.423359 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.423260    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:07:49.423359 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.423272    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:49.423359 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.423337    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:51.4233174 +0000 UTC m=+6.276887068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:49.605338 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.605242    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:02:48 +0000 UTC" deadline="2028-04-13 16:31:05.504372218 +0000 UTC"
Jul 23 00:07:49.605338 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.605286    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15136h23m15.899091711s"
Jul 23 00:07:49.733769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.732286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:49.733769 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:49.732489    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:49.760315 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.760277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5btk" event={"ID":"b6c68d68-0526-45e1-966a-883bb02511be","Type":"ContainerStarted","Data":"bfc0f35abf466176171bd4184380ad293ce973b992414d1f8e0510ab4b3e8426"}
Jul 23 00:07:49.776004 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.775947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgrtb" event={"ID":"15208ffd-1eda-488b-9099-d5133d306d2e","Type":"ContainerStarted","Data":"890afba8503153579aee3e0c9cfd04dd1edee9622131bd900e9778729a0957dc"}
Jul 23 00:07:49.782317 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.782281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7p5th" event={"ID":"d283f102-2e7d-4d9f-a493-aed1369cd78d","Type":"ContainerStarted","Data":"72af7d3237cdc7bd7bcbd9bf6af5c82a93a66931c18f73ec56937a25f326f2c0"}
Jul 23 00:07:49.791785 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.791633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"18ec1b058d6ff07e1abdf56dbf9195e08a5b1441c1252e362b58e3b9b58ce501"}
Jul 23 00:07:49.808386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.808120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bj99z" event={"ID":"a609c562-984b-4e40-a05d-58f2227174de","Type":"ContainerStarted","Data":"93885ea79b64b517ad31ea8fad6cff7a6634d6276476b400499e275cd897e34f"}
Jul 23 00:07:49.818451 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.818395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwthh" event={"ID":"629cb05f-3479-4f5b-a410-1c32ec4d52d1","Type":"ContainerStarted","Data":"87a50c868d3b4ccf69afdcb3f60d6ed996016646c6c9fa3317c063ebbe3f6523"}
Jul 23 00:07:49.823755 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.823726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerStarted","Data":"2a5d01919f85b6f9568556463b78e8967b5410e643551cb0250e86ed1d2dde43"}
Jul 23 00:07:49.829678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.828880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal" event={"ID":"642dc77bb1e95d4ad1afe2c58591ff7f","Type":"ContainerStarted","Data":"5d6e7ad2e510bb7ef6203adc37759a94bd54384494ebc2a571dd780629aec9da"}
Jul 23 00:07:49.843496 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.843419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsz4z" event={"ID":"e5bc1acf-579e-4827-bbc5-3877510d9f03","Type":"ContainerStarted","Data":"8cacc0c713cb4a05fed02b2bd4c836467eadeced6ed116fef4a36fd2b502fc76"}
Jul 23 00:07:49.850750 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.850678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv" event={"ID":"d2c52e54-4cd4-45d8-86ad-f2dc73069e38","Type":"ContainerStarted","Data":"0f4b3834e2708f7be8086b4c68d0471848e935b14268e9427721f053e896b491"}
Jul 23 00:07:49.861765 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.861145    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed36492455dccc8c0cba69df6878eb97" containerID="9354b0d75116d3b9be07d318d07cd778c72f58a1e7b01992e01f45fe8fe77c93" exitCode=0
Jul 23 00:07:49.861765 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.861191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal" event={"ID":"ed36492455dccc8c0cba69df6878eb97","Type":"ContainerDied","Data":"9354b0d75116d3b9be07d318d07cd778c72f58a1e7b01992e01f45fe8fe77c93"}
Jul 23 00:07:49.879680 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:49.878526    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-133.ec2.internal" podStartSLOduration=2.878507461 podStartE2EDuration="2.878507461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:07:49.847901356 +0000 UTC m=+4.701471036" watchObservedRunningTime="2026-07-23 00:07:49.878507461 +0000 UTC m=+4.732077138"
Jul 23 00:07:50.719791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:50.719757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:50.719967 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:50.719896    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:50.720025 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:50.719975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:50.720124 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:50.720081    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:50.898259 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:50.898174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal" event={"ID":"ed36492455dccc8c0cba69df6878eb97","Type":"ContainerStarted","Data":"beb05444bccca61f4a635732bbfe639a9353c290a2e29c6061ac8fc993406afa"}
Jul 23 00:07:50.917147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:50.916831    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-133.ec2.internal" podStartSLOduration=3.916808439 podStartE2EDuration="3.916808439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:07:50.915430902 +0000 UTC m=+5.769000580" watchObservedRunningTime="2026-07-23 00:07:50.916808439 +0000 UTC m=+5.770378118"
Jul 23 00:07:51.349521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:51.349434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:51.349521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:51.349494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:51.349755 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.349641    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:51.349755 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.349703    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:55.349684265 +0000 UTC m=+10.203253920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:51.350150 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.350128    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:51.350251 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.350191    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:55.350174321 +0000 UTC m=+10.203743978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:51.450743 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:51.450702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:51.450919 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.450872    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:07:51.450919 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.450892    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:07:51.450919 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.450908    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:51.451131 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.450970    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:07:55.450950859 +0000 UTC m=+10.304520515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:51.726124 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:51.725608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:51.726124 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:51.725742    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:52.720033 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:52.719976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:52.720548 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:52.720212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:52.720548 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:52.720351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:52.720548 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:52.720442    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:53.720025 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:53.719989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:53.720238 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:53.720143    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:54.719544 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:54.719501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:54.719739 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:54.719639    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:54.719912 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:54.719879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:54.720205 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:54.720159    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:55.384965 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:55.384926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:55.385502 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:55.385006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:55.385502 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.385177    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:55.385502 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.385231    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.385215047 +0000 UTC m=+18.238784700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:07:55.385684 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.385543    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:55.385684 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.385592    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.385580689 +0000 UTC m=+18.239150341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:07:55.485688 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:55.485648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:55.485878 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.485822    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:07:55.485878 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.485843    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:07:55.485878 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.485858    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:55.486056 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.485926    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:03.485907307 +0000 UTC m=+18.339476964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:07:55.721328 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:55.721249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:55.721494 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:55.721377    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:56.719599 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:56.719555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:56.720139 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:56.719776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:56.720139 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:56.719881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:56.720139 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:56.720001    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:57.720922 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:57.720583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:57.720922 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:57.720726    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:07:58.719477 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:58.719448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:07:58.719661 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:58.719453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:07:58.719661 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:58.719582    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:07:58.719661 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:58.719656    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:07:59.720833 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:07:59.720779    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:07:59.721309 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:07:59.720947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:00.719608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:00.719575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:00.719789 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:00.719713    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:00.719872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:00.719778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:00.719924 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:00.719893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:01.719795 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:01.719762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:01.720266 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:01.719893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:02.719926 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:02.719891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:02.720395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:02.719897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:02.720395 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:02.720053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:02.720395 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:02.720100    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:03.446663 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:03.446620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:03.446844 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:03.446715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:03.446844 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.446802    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:08:03.446939 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.446882    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:19.446862502 +0000 UTC m=+34.300432173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:08:03.446939 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.446809    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:08:03.447046 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.446985    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:19.446953419 +0000 UTC m=+34.300523070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:08:03.548037 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:03.547995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:03.548221 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.548196    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:08:03.548221 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.548219    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:08:03.548331 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.548232    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:08:03.548331 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.548299    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:19.548278506 +0000 UTC m=+34.401848174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:08:03.720028 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:03.719938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:03.720422 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:03.720090    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:04.720130 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:04.720093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:04.720524 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:04.720099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:04.720524 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:04.720198    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:04.720524 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:04.720325    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:05.721147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:05.721110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:05.721557 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:05.721221    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:06.720193 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:06.720154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:06.720193 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:06.720187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:06.720413 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:06.720297    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:06.720461 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:06.720446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:07.720159 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:07.719596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:07.720159 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:07.719745    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:08.720013 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.719824    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:08.720107 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.719868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:08.720143 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:08.720103    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:08.720175 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:08.720157    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:08.934701 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.934670    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="43543d177f231b9e264555ff330d7969ad5ad89031c3c93be145058e0c233d99" exitCode=0
Jul 23 00:08:08.935563 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.934768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"43543d177f231b9e264555ff330d7969ad5ad89031c3c93be145058e0c233d99"}
Jul 23 00:08:08.936692 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.936665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv" event={"ID":"d2c52e54-4cd4-45d8-86ad-f2dc73069e38","Type":"ContainerStarted","Data":"c38f2b477ccec566c3064ff890400673f326c5e7e8c2104ad1df8ad41a1258f1"}
Jul 23 00:08:08.938321 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.938293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5btk" event={"ID":"b6c68d68-0526-45e1-966a-883bb02511be","Type":"ContainerStarted","Data":"7ae8cec837ac66bb3fef6f854a7e5e172b22188135b18585a452f34550d25a18"}
Jul 23 00:08:08.940485 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.940254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgrtb" event={"ID":"15208ffd-1eda-488b-9099-d5133d306d2e","Type":"ContainerStarted","Data":"7c3bc5c873f666c7a88046571401551777edfbbed1165ffd3851dfdd6ee8ed9d"}
Jul 23 00:08:08.941836 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.941810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7p5th" event={"ID":"d283f102-2e7d-4d9f-a493-aed1369cd78d","Type":"ContainerStarted","Data":"a60de1e59795ee37c30f151be0bfa424e92b81619980696c520020cba87e9446"}
Jul 23 00:08:08.944865 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.944804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"5070a8b4f4c171d6b28579d4d2a4e84ad617c48bb91e276731351f75bfe4ef31"}
Jul 23 00:08:08.944986 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.944880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"9717d922bd3503989a814e504032360b9163c7058a110d5aa108380f4ab2da37"}
Jul 23 00:08:08.944986 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.944960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"bedbf8f04966605a28a158f4b48854a9197253c44d80475dc1fc89d405d57b18"}
Jul 23 00:08:08.945068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.945025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"45ae7219926f8ed04ba16da5d3ef255d5ecfc84c857e8e39bbd46046128c8b35"}
Jul 23 00:08:08.945068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.945064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"0ed0654903db927d9c2939000757417977ca880558e51fe85787b7e956a24511"}
Jul 23 00:08:08.946354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.946313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bj99z" event={"ID":"a609c562-984b-4e40-a05d-58f2227174de","Type":"ContainerStarted","Data":"c37f94044e6195be84558beb26361570cf4b55ff3064da0afd6d2355002db489"}
Jul 23 00:08:08.947759 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.947731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwthh" event={"ID":"629cb05f-3479-4f5b-a410-1c32ec4d52d1","Type":"ContainerStarted","Data":"60be927a6ae455c1ea8ad8a65a000b8078d3cfe575a07c2da0530cf4b91f3b2e"}
Jul 23 00:08:08.973851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.973802    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v5btk" podStartSLOduration=11.675192217 podStartE2EDuration="22.973785654s" totalImagesPullingTime="11.298593437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:08.973635948 +0000 UTC m=+23.827205617" watchObservedRunningTime="2026-07-23 00:08:08.973785654 +0000 UTC m=+23.827355337"
Jul 23 00:08:08.989242 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:08.989203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7p5th" podStartSLOduration=3.662106786 podStartE2EDuration="22.989189616s" totalImagesPullingTime="19.32708283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:08.988816481 +0000 UTC m=+23.842386154" watchObservedRunningTime="2026-07-23 00:08:08.989189616 +0000 UTC m=+23.842759287"
Jul 23 00:08:09.006544 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.006496    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sgrtb" podStartSLOduration=3.677734123 podStartE2EDuration="23.006482783s" totalImagesPullingTime="19.32874866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:09.006101119 +0000 UTC m=+23.859670794" watchObservedRunningTime="2026-07-23 00:08:09.006482783 +0000 UTC m=+23.860052479"
Jul 23 00:08:09.027680 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.027632    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bj99z" podStartSLOduration=3.600606966 podStartE2EDuration="23.027616741s" totalImagesPullingTime="19.427009775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:09.027163566 +0000 UTC m=+23.880733240" watchObservedRunningTime="2026-07-23 00:08:09.027616741 +0000 UTC m=+23.881186414"
Jul 23 00:08:09.043656 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.043559    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dwthh" podStartSLOduration=3.686721678 podStartE2EDuration="23.043545404s" totalImagesPullingTime="19.356823726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:09.043078851 +0000 UTC m=+23.896648525" watchObservedRunningTime="2026-07-23 00:08:09.043545404 +0000 UTC m=+23.897115076"
Jul 23 00:08:09.523951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.523920    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 00:08:09.703897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.703757    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:08:09.523945496Z","UUID":"02f487e2-6678-407a-a019-bab5602d5b91","Handler":null,"Name":"","Endpoint":""}
Jul 23 00:08:09.705496 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.705476    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 00:08:09.705634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.705514    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 00:08:09.719702 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.719674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:09.719843 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:09.719799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:09.954956 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.954869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"6ff417c39c438d1f77f8bb15106965ad4951550b56334b681814cfe6877e48b9"}
Jul 23 00:08:09.957720 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.957684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsz4z" event={"ID":"e5bc1acf-579e-4827-bbc5-3877510d9f03","Type":"ContainerStarted","Data":"28163609da15b77bb056076031f508a182cdd9cc6605af83ba909517fc6402d6"}
Jul 23 00:08:09.960140 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.960106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv" event={"ID":"d2c52e54-4cd4-45d8-86ad-f2dc73069e38","Type":"ContainerStarted","Data":"19791c0e0c510a1fb1105e1a2a928f5f102963ba899dcb6b1e883a2e84a07c0d"}
Jul 23 00:08:09.972076 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:09.972016    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qsz4z" podStartSLOduration=4.618876171 podStartE2EDuration="23.971996534s" totalImagesPullingTime="19.353120363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:09.971916967 +0000 UTC m=+24.825486643" watchObservedRunningTime="2026-07-23 00:08:09.971996534 +0000 UTC m=+24.825566208"
Jul 23 00:08:10.497015 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.496979    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:08:10.497597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.497575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:08:10.719834 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.719795    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:10.720068 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:10.719919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:10.720068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.719984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:10.720165 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:10.720092    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:10.964573 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.964492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv" event={"ID":"d2c52e54-4cd4-45d8-86ad-f2dc73069e38","Type":"ContainerStarted","Data":"af505c4c88b0db31903172ce7a5aacdd4a635071e4eac1189569d3e4c1c30113"}
Jul 23 00:08:10.965147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.964671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:08:10.965212 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.965188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v5btk"
Jul 23 00:08:10.982756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:10.982705    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lf4tv" podStartSLOduration=3.635756561 podStartE2EDuration="24.982690082s" totalImagesPullingTime="21.346933521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:10.982331476 +0000 UTC m=+25.835901153" watchObservedRunningTime="2026-07-23 00:08:10.982690082 +0000 UTC m=+25.836259756"
Jul 23 00:08:11.720258 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:11.720015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:11.720432 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:11.720361    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:11.970674 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:11.970586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"dc70fef8f9347cd534c2164cad6a0b89b12e43b80dfec37c9f5e4b9f52412896"}
Jul 23 00:08:12.719603 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:12.719564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:12.719783 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:12.719563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:12.719783 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:12.719709    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:12.719871 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:12.719799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:13.720031 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.719744    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:13.720780 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:13.720052    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:13.980117 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.980029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" event={"ID":"d0779a56-6596-4699-9aca-fa8d312ebda9","Type":"ContainerStarted","Data":"58d5b568b5424e2f56468e0d962b624cc362a134f187a3d987895e68d3957ab6"}
Jul 23 00:08:13.980333 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.980293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:13.981840 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.981817    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="63fddd6da6e660a5fc40d89df2b370f4e22136cc3e15bd483b1ee3adadba8ba4" exitCode=0
Jul 23 00:08:13.982153 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.981853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"63fddd6da6e660a5fc40d89df2b370f4e22136cc3e15bd483b1ee3adadba8ba4"}
Jul 23 00:08:13.997155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:13.997130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:14.006861 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:14.006818    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9" podStartSLOduration=8.580074887 podStartE2EDuration="28.006802146s" totalImagesPullingTime="19.426727259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:14.006532279 +0000 UTC m=+28.860101954" watchObservedRunningTime="2026-07-23 00:08:14.006802146 +0000 UTC m=+28.860371801"
Jul 23 00:08:14.719907 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:14.719868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:14.720190 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:14.719914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:14.720190 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:14.720048    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:14.720190 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:14.720167    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:14.984679 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:14.984608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:14.984845 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:14.984724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:15.000333 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.000308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:15.364521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.364490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j9hf9"]
Jul 23 00:08:15.364679 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.364603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:15.364726 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:15.364680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:15.368434 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.368402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-khbk4"]
Jul 23 00:08:15.368576 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.368518    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:15.368630 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:15.368609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:15.368999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.368961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vgc5"]
Jul 23 00:08:15.369119 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.369072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:15.369171 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:15.369151    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:15.988020 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.987989    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="7051e59fb1a3692bc3ea4ca4af0472a1fdf57c58ae778ffc8c40a47c10fde067" exitCode=0
Jul 23 00:08:15.988651 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:15.988073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"7051e59fb1a3692bc3ea4ca4af0472a1fdf57c58ae778ffc8c40a47c10fde067"}
Jul 23 00:08:16.720025 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:16.719817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:16.720202 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:16.719818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:16.720202 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:16.720106    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:16.720202 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:16.719818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:16.720330 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:16.720201    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:16.720330 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:16.720245    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:17.994495 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:17.994455    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="f4e815eb79e637672d1d40d2ef3da46d669d28735a1d8b1d11c62b8267e76244" exitCode=0
Jul 23 00:08:17.995027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:17.994530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"f4e815eb79e637672d1d40d2ef3da46d669d28735a1d8b1d11c62b8267e76244"}
Jul 23 00:08:18.720518 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:18.720485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:18.720687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:18.720526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:18.720687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:18.720525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:18.720687 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:18.720617    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:18.720853 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:18.720729    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:18.720853 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:18.720807    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:19.475257 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:19.475210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:19.475958 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:19.475266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:19.475958 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.475374    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 00:08:19.475958 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.475449    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret podName:ce30ed35-f91a-40eb-992a-ed17e7a1bd30 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:51.475430818 +0000 UTC m=+66.329000477 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret") pod "global-pull-secret-syncer-j9hf9" (UID: "ce30ed35-f91a-40eb-992a-ed17e7a1bd30") : object "kube-system"/"original-pull-secret" not registered
Jul 23 00:08:19.475958 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.475381    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:08:19.475958 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.475522    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:51.475504875 +0000 UTC m=+66.329074532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 00:08:19.575696 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:19.575662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:19.575881 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.575850    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 00:08:19.575881 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.575878    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 00:08:19.576031 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.575890    2565 projected.go:196] Error preparing data for projected volume kube-api-access-knkwk for pod openshift-network-diagnostics/network-check-target-khbk4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:08:19.576031 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:19.575946    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk podName:a44c815c-a906-4d20-b102-daf16942da61 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:51.575929176 +0000 UTC m=+66.429498851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-knkwk" (UniqueName: "kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk") pod "network-check-target-khbk4" (UID: "a44c815c-a906-4d20-b102-daf16942da61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 00:08:20.720581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:20.720484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:20.720581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:20.720518    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:20.720581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:20.720518    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:20.721296 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:20.720618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-khbk4" podUID="a44c815c-a906-4d20-b102-daf16942da61"
Jul 23 00:08:20.721296 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:20.720682    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j9hf9" podUID="ce30ed35-f91a-40eb-992a-ed17e7a1bd30"
Jul 23 00:08:20.721296 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:20.720775    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:08:21.967077 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:21.967048    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-133.ec2.internal" event="NodeReady"
Jul 23 00:08:21.967568 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:21.967207    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 00:08:22.000819 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.000790    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f9857955-kmxsw"]
Jul 23 00:08:22.003422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.003393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.005905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.005887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 00:08:22.006062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.006009    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 00:08:22.006062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.006035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 00:08:22.006434 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.006418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t8qkr\""
Jul 23 00:08:22.012702 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.012680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 00:08:22.017498 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.017472    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"]
Jul 23 00:08:22.019861 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.019833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.020270 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.020242    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"]
Jul 23 00:08:22.022449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 00:08:22.022727 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022706    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"]
Jul 23 00:08:22.022825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 00:08:22.022825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022800    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.022939 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 00:08:22.023018 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.022838    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 00:08:22.025535 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.025505    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"]
Jul 23 00:08:22.028598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.026242    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 00:08:22.028598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.026268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 00:08:22.028598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.026507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.028598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.026250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 00:08:22.028598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.027844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 00:08:22.029918 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.029879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f9857955-kmxsw"]
Jul 23 00:08:22.030028 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.030013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.030082 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.030036    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-thxht"]
Jul 23 00:08:22.031746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.031491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 23 00:08:22.031746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.031511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 23 00:08:22.031746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.031521    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t88zl\""
Jul 23 00:08:22.031954 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.031835    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r8cbw"]
Jul 23 00:08:22.032050 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.032032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.034626 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.034606    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 00:08:22.035085 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.034948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-gkqvn\""
Jul 23 00:08:22.035191 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.035138    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 00:08:22.035382 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.035363    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-httr9\""
Jul 23 00:08:22.035624 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.035604    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 00:08:22.035813 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.035798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 00:08:22.037332 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.037309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"]
Jul 23 00:08:22.038039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.037349    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"]
Jul 23 00:08:22.038039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.037373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"]
Jul 23 00:08:22.038039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.037387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"]
Jul 23 00:08:22.038039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.037473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.041717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.041693    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 00:08:22.041717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.041707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\""
Jul 23 00:08:22.041938 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.041921    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 00:08:22.050630 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.050438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thxht"]
Jul 23 00:08:22.051350 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.051328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8cbw"]
Jul 23 00:08:22.092456 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-klusterlet-config\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.092588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.092588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21cfc68c-5751-42f7-a018-383e03338cd7-config-volume\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.092773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcvv\" (UniqueName: \"kubernetes.io/projected/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-kube-api-access-dwcvv\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.092773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/561926cd-363d-4471-a8ea-f1a3878284aa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.092773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8b92\" (UniqueName: \"kubernetes.io/projected/21cfc68c-5751-42f7-a018-383e03338cd7-kube-api-access-l8b92\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.092951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3101d750-8df5-4897-9428-b552c33a6c74-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.092951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjhb\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.092951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.092928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-cert\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.093129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.093129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfc68c-5751-42f7-a018-383e03338cd7-tmp-dir\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.093129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfpb\" (UniqueName: \"kubernetes.io/projected/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-kube-api-access-bqfpb\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.093129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.093269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvks\" (UniqueName: \"kubernetes.io/projected/5d0a5061-c41e-46f4-8f71-1022202698b9-kube-api-access-rqvks\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.093269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.093269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d0a5061-c41e-46f4-8f71-1022202698b9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.093269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.093269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.093442 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-tmp\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.093442 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.093442 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.093365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/561926cd-363d-4471-a8ea-f1a3878284aa-kube-api-access-5rsrm\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.173140 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.173105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v5btk_b6c68d68-0526-45e1-966a-883bb02511be/konnectivity-agent/0.log" limit=1024
Jul 23 00:08:22.194232 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.194232 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-tmp\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/561926cd-363d-4471-a8ea-f1a3878284aa-kube-api-access-5rsrm\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.194346    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-klusterlet-config\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.194421    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:22.694400795 +0000 UTC m=+37.547970448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:22.194462 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21cfc68c-5751-42f7-a018-383e03338cd7-config-volume\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcvv\" (UniqueName: \"kubernetes.io/projected/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-kube-api-access-dwcvv\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/561926cd-363d-4471-a8ea-f1a3878284aa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8b92\" (UniqueName: \"kubernetes.io/projected/21cfc68c-5751-42f7-a018-383e03338cd7-kube-api-access-l8b92\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3101d750-8df5-4897-9428-b552c33a6c74-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.194782 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjhb\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-cert\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfc68c-5751-42f7-a018-383e03338cd7-tmp-dir\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfpb\" (UniqueName: \"kubernetes.io/projected/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-kube-api-access-bqfpb\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.194964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvks\" (UniqueName: \"kubernetes.io/projected/5d0a5061-c41e-46f4-8f71-1022202698b9-kube-api-access-rqvks\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d0a5061-c41e-46f4-8f71-1022202698b9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.195267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.195304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-tmp\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.195410    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.195426    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.195486    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:22.695468214 +0000 UTC m=+37.549037880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.195597    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:22.195825 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.195651    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:22.695635254 +0000 UTC m=+37.549204916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:22.196203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.196046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/21cfc68c-5751-42f7-a018-383e03338cd7-config-volume\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.196203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.196073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/561926cd-363d-4471-a8ea-f1a3878284aa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.196203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.196112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3101d750-8df5-4897-9428-b552c33a6c74-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.196484 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.196465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfc68c-5751-42f7-a018-383e03338cd7-tmp-dir\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.200555 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.200528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-cert\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.200677 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.200580    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.200746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.200716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-ca\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.201490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.200908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-klusterlet-config\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.201490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.201248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.201490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.201323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.201490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.201446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5d0a5061-c41e-46f4-8f71-1022202698b9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.201490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.201450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/561926cd-363d-4471-a8ea-f1a3878284aa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.201868 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.201843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.203787 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.203767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcvv\" (UniqueName: \"kubernetes.io/projected/a98c84a3-e7bf-451f-84bb-d0a6914fb9d6-kube-api-access-dwcvv\") pod \"ingress-canary-thxht\" (UID: \"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6\") " pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.203897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.203868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8b92\" (UniqueName: \"kubernetes.io/projected/21cfc68c-5751-42f7-a018-383e03338cd7-kube-api-access-l8b92\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.206917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.206872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfpb\" (UniqueName: \"kubernetes.io/projected/d4e9d0db-3712-4daf-93e2-230aa9ba8e99-kube-api-access-bqfpb\") pod \"klusterlet-addon-workmgr-78895ddc9c-nschw\" (UID: \"d4e9d0db-3712-4daf-93e2-230aa9ba8e99\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.208506 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.208232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsrm\" (UniqueName: \"kubernetes.io/projected/561926cd-363d-4471-a8ea-f1a3878284aa-kube-api-access-5rsrm\") pod \"cluster-proxy-proxy-agent-78f9598457-bbh8x\" (UID: \"561926cd-363d-4471-a8ea-f1a3878284aa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.209077 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.209055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjhb\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.213011 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.212950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.213211 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.213187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvks\" (UniqueName: \"kubernetes.io/projected/5d0a5061-c41e-46f4-8f71-1022202698b9-kube-api-access-rqvks\") pod \"managed-serviceaccount-addon-agent-54546bc5c6-vcrpg\" (UID: \"5d0a5061-c41e-46f4-8f71-1022202698b9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.273018 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.272919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-v5btk_b6c68d68-0526-45e1-966a-883bb02511be/konnectivity-agent/0.log" limit=1024
Jul 23 00:08:22.338033 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.337993    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:22.354731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.354508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:08:22.368928 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.368893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:08:22.383610 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.383540    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-thxht"
Jul 23 00:08:22.525751 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.525636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"]
Jul 23 00:08:22.533489 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.533427    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"]
Jul 23 00:08:22.545735 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.545575    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"]
Jul 23 00:08:22.560807 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.560775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-thxht"]
Jul 23 00:08:22.576241 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:08:22.576210    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d0a5061_c41e_46f4_8f71_1022202698b9.slice/crio-0a527d2d4ed9b765918482d4ec97fd46ebcb68c7e7ad38f8bb51e792567fb720 WatchSource:0}: Error finding container 0a527d2d4ed9b765918482d4ec97fd46ebcb68c7e7ad38f8bb51e792567fb720: Status 404 returned error can't find the container with id 0a527d2d4ed9b765918482d4ec97fd46ebcb68c7e7ad38f8bb51e792567fb720
Jul 23 00:08:22.584509 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:08:22.584409    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98c84a3_e7bf_451f_84bb_d0a6914fb9d6.slice/crio-deeb7eb758d4cb8ba7bc041c7a26f47535ac85555b3b648e33a67576ee989fff WatchSource:0}: Error finding container deeb7eb758d4cb8ba7bc041c7a26f47535ac85555b3b648e33a67576ee989fff: Status 404 returned error can't find the container with id deeb7eb758d4cb8ba7bc041c7a26f47535ac85555b3b648e33a67576ee989fff
Jul 23 00:08:22.700129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.700096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:22.700301 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.700139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:22.700301 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.700166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:22.700301 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700258    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:22.700301 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700272    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:22.700446 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700327    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:23.700315209 +0000 UTC m=+38.553884859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:22.700446 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700258    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:22.700446 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700398    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:23.700385397 +0000 UTC m=+38.553955047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:22.700446 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700258    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:22.700446 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:22.700432    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:23.700424745 +0000 UTC m=+38.553994396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:22.720461 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.720427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:22.720635 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.720427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:22.720635 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.720431    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:22.723195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723169    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\""
Jul 23 00:08:22.723195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\""
Jul 23 00:08:22.723440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723424    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 00:08:22.723571 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723543    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 00:08:22.723663 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 00:08:22.723663 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:22.723551    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 00:08:23.008417 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.008380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerStarted","Data":"59152495407e542d4fc70391e526c7b119ba420e963d207df01235e93d0653c6"}
Jul 23 00:08:23.009393 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.009365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thxht" event={"ID":"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6","Type":"ContainerStarted","Data":"deeb7eb758d4cb8ba7bc041c7a26f47535ac85555b3b648e33a67576ee989fff"}
Jul 23 00:08:23.010420 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.010397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerStarted","Data":"0a527d2d4ed9b765918482d4ec97fd46ebcb68c7e7ad38f8bb51e792567fb720"}
Jul 23 00:08:23.012845 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.012824    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="a0b2e329b249ad9c5ca1523e8d4dd683b869cec110de2e17933a313c824506f1" exitCode=0
Jul 23 00:08:23.012958 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.012885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"a0b2e329b249ad9c5ca1523e8d4dd683b869cec110de2e17933a313c824506f1"}
Jul 23 00:08:23.014303 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.013875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerStarted","Data":"5d2ebf7e1a9791ea63d42e57737fd981ebdc624b396ea66c993242cc20ee2cad"}
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.709270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.709597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:23.709651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.709845    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.709862    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.709921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:25.709902895 +0000 UTC m=+40.563472547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.710407    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.710461    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:25.710445676 +0000 UTC m=+40.564015327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.710524    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:23.710590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:23.710557    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:25.710546439 +0000 UTC m=+40.564116090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:24.031520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:24.030396    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c736f2d-3dc5-4f07-963c-c5deb04f105f" containerID="eb61c1070f2c4072e8e957534f903cb460115d93e1942265c8462309c42f89fa" exitCode=0
Jul 23 00:08:24.031520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:24.030465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerDied","Data":"eb61c1070f2c4072e8e957534f903cb460115d93e1942265c8462309c42f89fa"}
Jul 23 00:08:25.045908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:25.045155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" event={"ID":"3c736f2d-3dc5-4f07-963c-c5deb04f105f","Type":"ContainerStarted","Data":"ec1be0b4e48a37bd4b8027c06aed3a7873500b36c07b7b66f3a0f6c9d0926339"}
Jul 23 00:08:25.071791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:25.071214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fxk7c" podStartSLOduration=8.868400377 podStartE2EDuration="39.071191217s" totalImagesPullingTime="30.20279084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:25.069042047 +0000 UTC m=+39.922611723" watchObservedRunningTime="2026-07-23 00:08:25.071191217 +0000 UTC m=+39.924760892"
Jul 23 00:08:25.728955 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:25.728916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:25.729145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:25.728989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:25.729145 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729086    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:25.729145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:25.729111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:25.729326 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729152    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:29.729133237 +0000 UTC m=+44.582702902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:25.729326 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729227    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:25.729326 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729276    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:29.729260108 +0000 UTC m=+44.582829762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:25.729479 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729341    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:25.729479 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729352    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:25.729479 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:25.729384    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:29.729373806 +0000 UTC m=+44.582943460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:29.767141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:29.767098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:29.767141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:29.767155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:29.767189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767272    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767304    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767308    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767362    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767364    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:37.767343785 +0000 UTC m=+52.620913445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767386    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:37.767376124 +0000 UTC m=+52.620945777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:29.767784 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:29.767403    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:37.767393577 +0000 UTC m=+52.620963231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:31.063336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.063256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerStarted","Data":"b89ce564600ee830f57d332ca60c6e200f64c0d042edf5cf8926fc2f395a8d90"}
Jul 23 00:08:31.063336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.063296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerStarted","Data":"6f4a54a7ff0d3431f8751a08b467c0d9ef7c3b7bb9dbd1b83e11046e3570c776"}
Jul 23 00:08:31.063336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.063315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerStarted","Data":"6f3ea829f8fe194d4ad0ae4aa069085898bbdec03b5efa119e5426869cb54d12"}
Jul 23 00:08:31.064744 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.064722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-thxht" event={"ID":"a98c84a3-e7bf-451f-84bb-d0a6914fb9d6","Type":"ContainerStarted","Data":"fa6b72311fd07607df8bb8ef7bb3c98efbd9963f489669c51c00ac3a464b4a1b"}
Jul 23 00:08:31.066148 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.066121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerStarted","Data":"950823190a78b9e39bc0294527b4b2ae6ce044d7c61fa50d489a2038ec4c0a19"}
Jul 23 00:08:31.067458 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.067438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerStarted","Data":"95d5660449b43f49c74b60d1f15b9ee082727c505bf2154d3c3ca81bb70b029a"}
Jul 23 00:08:31.067633 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.067620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:31.069179 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.069159    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:08:31.083031 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.082981    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" podStartSLOduration=17.075138811 podStartE2EDuration="25.082953282s" totalImagesPullingTime="8.007814471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:31.081853158 +0000 UTC m=+45.935422832" watchObservedRunningTime="2026-07-23 00:08:31.082953282 +0000 UTC m=+45.936522933"
Jul 23 00:08:31.097338 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.097293    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" podStartSLOduration=17.105102637 podStartE2EDuration="25.097280636s" totalImagesPullingTime="7.992177999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:31.096281228 +0000 UTC m=+45.949850900" watchObservedRunningTime="2026-07-23 00:08:31.097280636 +0000 UTC m=+45.950850309"
Jul 23 00:08:31.118143 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.118100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" podStartSLOduration=17.118257432 podStartE2EDuration="25.11808547s" totalImagesPullingTime="7.999828038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:31.117740607 +0000 UTC m=+45.971310280" watchObservedRunningTime="2026-07-23 00:08:31.11808547 +0000 UTC m=+45.971655121"
Jul 23 00:08:31.134622 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:31.134573    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-thxht" podStartSLOduration=1.141964517 podStartE2EDuration="9.134557655s" totalImagesPullingTime="7.992593138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:31.133872153 +0000 UTC m=+45.987441826" watchObservedRunningTime="2026-07-23 00:08:31.134557655 +0000 UTC m=+45.988127328"
Jul 23 00:08:37.842141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:37.842103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:37.842141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:37.842146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:37.842169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842260    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842338    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:53.842323188 +0000 UTC m=+68.695892840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842260    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842405    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:08:53.842394091 +0000 UTC m=+68.695963745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842261    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842423    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:37.842563 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:37.842453    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:08:53.842445958 +0000 UTC m=+68.696015609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:47.009870 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:47.009842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghsx9"
Jul 23 00:08:51.549592 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.549545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:51.549592 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.549594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:08:51.552664 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.552643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 00:08:51.552760 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.552714    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 00:08:51.560729 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:51.560707    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:08:51.560818 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:51.560765    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:55.560749994 +0000 UTC m=+130.414319646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : secret "metrics-daemon-secret" not found
Jul 23 00:08:51.564435 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.564414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce30ed35-f91a-40eb-992a-ed17e7a1bd30-original-pull-secret\") pod \"global-pull-secret-syncer-j9hf9\" (UID: \"ce30ed35-f91a-40eb-992a-ed17e7a1bd30\") " pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:51.650570 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.650534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:51.653805 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.653786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 00:08:51.663628 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.663608    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 00:08:51.673741 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.673718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkwk\" (UniqueName: \"kubernetes.io/projected/a44c815c-a906-4d20-b102-daf16942da61-kube-api-access-knkwk\") pod \"network-check-target-khbk4\" (UID: \"a44c815c-a906-4d20-b102-daf16942da61\") " pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:51.837465 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.837376    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j9hf9"
Jul 23 00:08:51.845067 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.845048    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8bbv\""
Jul 23 00:08:51.852199 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.852176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:51.963293 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.963266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j9hf9"]
Jul 23 00:08:51.965370 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:08:51.965342    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce30ed35_f91a_40eb_992a_ed17e7a1bd30.slice/crio-c8dae608e4bda0710a914aa5e300a8fdff1588d5308adb2ae6b9c38053924ebf WatchSource:0}: Error finding container c8dae608e4bda0710a914aa5e300a8fdff1588d5308adb2ae6b9c38053924ebf: Status 404 returned error can't find the container with id c8dae608e4bda0710a914aa5e300a8fdff1588d5308adb2ae6b9c38053924ebf
Jul 23 00:08:51.980688 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:51.980664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-khbk4"]
Jul 23 00:08:51.983161 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:08:51.983139    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44c815c_a906_4d20_b102_daf16942da61.slice/crio-3b6fb4dd17627293ae7a4409eb520271cc832dea11a367f19333ac728564b5dd WatchSource:0}: Error finding container 3b6fb4dd17627293ae7a4409eb520271cc832dea11a367f19333ac728564b5dd: Status 404 returned error can't find the container with id 3b6fb4dd17627293ae7a4409eb520271cc832dea11a367f19333ac728564b5dd
Jul 23 00:08:52.125280 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:52.125195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-khbk4" event={"ID":"a44c815c-a906-4d20-b102-daf16942da61","Type":"ContainerStarted","Data":"3b6fb4dd17627293ae7a4409eb520271cc832dea11a367f19333ac728564b5dd"}
Jul 23 00:08:52.126270 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:52.126247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j9hf9" event={"ID":"ce30ed35-f91a-40eb-992a-ed17e7a1bd30","Type":"ContainerStarted","Data":"c8dae608e4bda0710a914aa5e300a8fdff1588d5308adb2ae6b9c38053924ebf"}
Jul 23 00:08:53.870212 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:53.870179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:53.870231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:53.870265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870338    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870390    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870410    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:25.870395279 +0000 UTC m=+100.723964930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870429    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870439    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:09:25.870425524 +0000 UTC m=+100.723995175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870444    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:08:53.870655 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:08:53.870495    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:09:25.870479786 +0000 UTC m=+100.724049451 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:08:57.146428 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:57.146384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j9hf9" event={"ID":"ce30ed35-f91a-40eb-992a-ed17e7a1bd30","Type":"ContainerStarted","Data":"a6d8d2d6dac0eef6da1b334778c2010dc55ae7ca690e4f43cc07932c7c4b9583"}
Jul 23 00:08:57.147778 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:57.147757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-khbk4" event={"ID":"a44c815c-a906-4d20-b102-daf16942da61","Type":"ContainerStarted","Data":"4a0add6d070563c4c0ed512ce6ff156c93f67550f7f20cd2ac9b06235329271c"}
Jul 23 00:08:57.147909 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:57.147890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:08:57.161247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:57.161203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j9hf9" podStartSLOduration=65.615997098 podStartE2EDuration="1m10.161192435s" totalImagesPullingTime="4.545195337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:57.161172577 +0000 UTC m=+72.014742251" watchObservedRunningTime="2026-07-23 00:08:57.161192435 +0000 UTC m=+72.014762108"
Jul 23 00:08:57.186188 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:08:57.186145    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-khbk4" podStartSLOduration=66.661569837 podStartE2EDuration="1m11.186132804s" totalImagesPullingTime="4.524562967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:08:57.185817606 +0000 UTC m=+72.039387278" watchObservedRunningTime="2026-07-23 00:08:57.186132804 +0000 UTC m=+72.039702477"
Jul 23 00:09:25.902271 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:09:25.902238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:09:25.902278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:09:25.902381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902405    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902479    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:10:29.90245601 +0000 UTC m=+164.756025676 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902480    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902492    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902506    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902538    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:10:29.902524601 +0000 UTC m=+164.756094252 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:09:25.902700 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:25.902552    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:10:29.90254558 +0000 UTC m=+164.756115232 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:09:28.153860 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:09:28.153830    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-khbk4"
Jul 23 00:09:55.624432 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:09:55.624387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:09:55.624910 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:55.624536    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:09:55.624910 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:09:55.624600    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:11:57.624584815 +0000 UTC m=+252.478154471 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : secret "metrics-daemon-secret" not found
Jul 23 00:10:25.014706 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:25.014671    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-84f9857955-kmxsw" podUID="c7b89a7d-00ee-4317-846c-6bf510be494d"
Jul 23 00:10:25.061783 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:25.061736    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk" podUID="3101d750-8df5-4897-9428-b552c33a6c74"
Jul 23 00:10:25.088936 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:25.088907    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r8cbw" podUID="21cfc68c-5751-42f7-a018-383e03338cd7"
Jul 23 00:10:25.396387 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:25.396356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:10:25.396577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:25.396365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:10:25.396577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:25.396364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:10:25.731143 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:25.731062    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:10:29.976570 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:29.976534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:10:29.976570 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:29.976574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:29.976596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") pod \"image-registry-84f9857955-kmxsw\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") " pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976683    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976697    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9857955-kmxsw: secret "image-registry-tls" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976740    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls podName:c7b89a7d-00ee-4317-846c-6bf510be494d nodeName:}" failed. No retries permitted until 2026-07-23 00:12:31.976727426 +0000 UTC m=+286.830297076 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls") pod "image-registry-84f9857955-kmxsw" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d") : secret "image-registry-tls" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976681    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976814    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert podName:3101d750-8df5-4897-9428-b552c33a6c74 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:31.976800814 +0000 UTC m=+286.830370464 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5kjdk" (UID: "3101d750-8df5-4897-9428-b552c33a6c74") : secret "networking-console-plugin-cert" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976681    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 00:10:29.977042 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:10:29.976844    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls podName:21cfc68c-5751-42f7-a018-383e03338cd7 nodeName:}" failed. No retries permitted until 2026-07-23 00:12:31.976837199 +0000 UTC m=+286.830406850 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls") pod "dns-default-r8cbw" (UID: "21cfc68c-5751-42f7-a018-383e03338cd7") : secret "dns-default-metrics-tls" not found
Jul 23 00:10:31.068515 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.068422    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" podUID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/readyz\": dial tcp 10.132.0.7:8000: connect: connection refused"
Jul 23 00:10:31.414479 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.414395    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d0a5061-c41e-46f4-8f71-1022202698b9" containerID="950823190a78b9e39bc0294527b4b2ae6ce044d7c61fa50d489a2038ec4c0a19" exitCode=255
Jul 23 00:10:31.414479 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.414465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerDied","Data":"950823190a78b9e39bc0294527b4b2ae6ce044d7c61fa50d489a2038ec4c0a19"}
Jul 23 00:10:31.415140 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.414822    2565 scope.go:117] "RemoveContainer" containerID="950823190a78b9e39bc0294527b4b2ae6ce044d7c61fa50d489a2038ec4c0a19"
Jul 23 00:10:31.418295 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.418244    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99" containerID="95d5660449b43f49c74b60d1f15b9ee082727c505bf2154d3c3ca81bb70b029a" exitCode=1
Jul 23 00:10:31.418364 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.418316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerDied","Data":"95d5660449b43f49c74b60d1f15b9ee082727c505bf2154d3c3ca81bb70b029a"}
Jul 23 00:10:31.418590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:31.418576    2565 scope.go:117] "RemoveContainer" containerID="95d5660449b43f49c74b60d1f15b9ee082727c505bf2154d3c3ca81bb70b029a"
Jul 23 00:10:32.339147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.339094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:10:32.369687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.369658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:10:32.423560 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.423529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerStarted","Data":"68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba"}
Jul 23 00:10:32.425115 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.425086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerStarted","Data":"db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"}
Jul 23 00:10:32.425294 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.425281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:10:32.426081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:32.426061    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:10:39.719952 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:10:39.719857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:11:57.698422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:11:57.698381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:11:57.699008 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:11:57.698515    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 00:11:57.699008 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:11:57.698588    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs podName:3ac5aa4f-7a25-4e1c-9f7a-579072177156 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:59.698567586 +0000 UTC m=+374.552137243 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs") pod "network-metrics-daemon-8vgc5" (UID: "3ac5aa4f-7a25-4e1c-9f7a-579072177156") : secret "metrics-daemon-secret" not found
Jul 23 00:12:04.856454 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.856418    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-56xk7"]
Jul 23 00:12:04.858424 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.858408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:04.867326 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.867304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 00:12:04.869071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.869049    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 00:12:04.869071 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.869069    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5btmf\""
Jul 23 00:12:04.869209 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.869052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 00:12:04.870091 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.870078    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 00:12:04.907698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.907664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56xk7"]
Jul 23 00:12:04.957039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.957008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfxv\" (UniqueName: \"kubernetes.io/projected/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-api-access-sdfxv\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:04.957191 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.957057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/481ac944-2252-4ac5-ba03-4d641edf2a2a-crio-socket\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:04.957191 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.957109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:04.957191 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.957181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:04.957290 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:04.957232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/481ac944-2252-4ac5-ba03-4d641edf2a2a-data-volume\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058400 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfxv\" (UniqueName: \"kubernetes.io/projected/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-api-access-sdfxv\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/481ac944-2252-4ac5-ba03-4d641edf2a2a-crio-socket\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/481ac944-2252-4ac5-ba03-4d641edf2a2a-data-volume\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/481ac944-2252-4ac5-ba03-4d641edf2a2a-crio-socket\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.058790 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.058603    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:05.058790 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.058626    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:12:05.058790 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.058665    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.558643337 +0000 UTC m=+260.412212990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:05.058790 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.058694    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:05.558677275 +0000 UTC m=+260.412246940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : configmap "kube-rbac-proxy" not found
Jul 23 00:12:05.058926 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.058880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/481ac944-2252-4ac5-ba03-4d641edf2a2a-data-volume\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.066484 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.066466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfxv\" (UniqueName: \"kubernetes.io/projected/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-api-access-sdfxv\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.562733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.562698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.562901 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:05.562745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:05.562901 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.562817    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:12:05.562901 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.562843    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:05.562901 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.562864    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.562851537 +0000 UTC m=+261.416421188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : configmap "kube-rbac-proxy" not found
Jul 23 00:12:05.563068 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:05.562903    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:06.562890992 +0000 UTC m=+261.416460644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:06.568885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:06.568851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:06.569276 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:06.568896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:06.569276 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:06.568982    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:12:06.569276 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:06.569012    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:06.569276 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:06.569031    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.569018127 +0000 UTC m=+263.422587781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : configmap "kube-rbac-proxy" not found
Jul 23 00:12:06.569276 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:06.569078    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:08.56906447 +0000 UTC m=+263.422634120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:06.850370 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:06.850296    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwthh_629cb05f-3479-4f5b-a410-1c32ec4d52d1/dns-node-resolver/0.log"
Jul 23 00:12:07.651186 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:07.651158    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7p5th_d283f102-2e7d-4d9f-a493-aed1369cd78d/node-ca/0.log"
Jul 23 00:12:08.586924 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:08.586890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:08.587125 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:08.586982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:08.587125 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:08.587021    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:12:08.587125 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:08.587076    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:08.587125 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:08.587083    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.587067877 +0000 UTC m=+267.440637532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : configmap "kube-rbac-proxy" not found
Jul 23 00:12:08.587125 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:08.587114    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:12.587103637 +0000 UTC m=+267.440673287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:08.850198 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:08.850117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thxht_a98c84a3-e7bf-451f-84bb-d0a6914fb9d6/serve-healthcheck-canary/0.log"
Jul 23 00:12:12.619084 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:12.619045    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:12.619503 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:12.619100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:12.619503 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:12.619194    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:12.619503 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:12.619235    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 00:12:12.619503 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:12.619274    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.619258273 +0000 UTC m=+275.472827924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:12.619503 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:12.619293    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:20.619280654 +0000 UTC m=+275.472850309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : configmap "kube-rbac-proxy" not found
Jul 23 00:12:20.679428 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:20.679369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:20.679895 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:20.679454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:20.679895 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:20.679530    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:20.679895 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:20.679614    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls podName:481ac944-2252-4ac5-ba03-4d641edf2a2a nodeName:}" failed. No retries permitted until 2026-07-23 00:12:36.679594157 +0000 UTC m=+291.533163831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-56xk7" (UID: "481ac944-2252-4ac5-ba03-4d641edf2a2a") : secret "insights-runtime-extractor-tls" not found
Jul 23 00:12:20.680092 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:20.680074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/481ac944-2252-4ac5-ba03-4d641edf2a2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:28.397741 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:28.397675    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-r8cbw" podUID="21cfc68c-5751-42f7-a018-383e03338cd7"
Jul 23 00:12:28.397741 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:28.397687    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk" podUID="3101d750-8df5-4897-9428-b552c33a6c74"
Jul 23 00:12:28.397741 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:28.397706    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-84f9857955-kmxsw" podUID="c7b89a7d-00ee-4317-846c-6bf510be494d"
Jul 23 00:12:28.762197 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:28.762173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:12:28.762197 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:28.762189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:28.762381 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:28.762175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:12:29.622953 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.622921    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f9857955-kmxsw"]
Jul 23 00:12:29.623349 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:29.623135    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-84f9857955-kmxsw" podUID="c7b89a7d-00ee-4317-846c-6bf510be494d"
Jul 23 00:12:29.765275 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.765247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:12:29.770520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.770499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:12:29.845125 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtjhb\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets\") pod \"c7b89a7d-00ee-4317-846c-6bf510be494d\" (UID: \"c7b89a7d-00ee-4317-846c-6bf510be494d\") "
Jul 23 00:12:29.845634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845583    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:12:29.845634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845609    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:12:29.845823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845798    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7b89a7d-00ee-4317-846c-6bf510be494d-ca-trust-extracted\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.845823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.845825    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-trusted-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.846139 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.846113    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:12:29.847837 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.847808    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:12:29.847920 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.847879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:12:29.847984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.847924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb" (OuterVolumeSpecName: "kube-api-access-dtjhb") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "kube-api-access-dtjhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:12:29.847984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.847960    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c7b89a7d-00ee-4317-846c-6bf510be494d" (UID: "c7b89a7d-00ee-4317-846c-6bf510be494d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:12:29.947195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.947124    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-certificates\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.947195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.947150    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-installation-pull-secrets\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.947195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.947160    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-bound-sa-token\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.947195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.947169    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjhb\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-kube-api-access-dtjhb\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:29.947195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:29.947178    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7b89a7d-00ee-4317-846c-6bf510be494d-image-registry-private-configuration\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:30.768204 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:30.768173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9857955-kmxsw"
Jul 23 00:12:30.803210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:30.803180    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f9857955-kmxsw"]
Jul 23 00:12:30.808866 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:30.808842    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f9857955-kmxsw"]
Jul 23 00:12:30.952756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:30.952729    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7b89a7d-00ee-4317-846c-6bf510be494d-registry-tls\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:12:31.726571 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.724385    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b89a7d-00ee-4317-846c-6bf510be494d" path="/var/lib/kubelet/pods/c7b89a7d-00ee-4317-846c-6bf510be494d/volumes"
Jul 23 00:12:31.772984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.772888    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d0a5061-c41e-46f4-8f71-1022202698b9" containerID="68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba" exitCode=255
Jul 23 00:12:31.773415 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.772958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerDied","Data":"68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba"}
Jul 23 00:12:31.773415 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.773035    2565 scope.go:117] "RemoveContainer" containerID="950823190a78b9e39bc0294527b4b2ae6ce044d7c61fa50d489a2038ec4c0a19"
Jul 23 00:12:31.773415 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.773324    2565 scope.go:117] "RemoveContainer" containerID="68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba"
Jul 23 00:12:31.773590 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:31.773521    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-54546bc5c6-vcrpg_open-cluster-management-agent-addon(5d0a5061-c41e-46f4-8f71-1022202698b9)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" podUID="5d0a5061-c41e-46f4-8f71-1022202698b9"
Jul 23 00:12:31.774673 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.774656    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99" containerID="db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd" exitCode=1
Jul 23 00:12:31.774747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.774688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerDied","Data":"db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"}
Jul 23 00:12:31.774918 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.774906    2565 scope.go:117] "RemoveContainer" containerID="db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"
Jul 23 00:12:31.775072 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:31.775055    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-78895ddc9c-nschw_open-cluster-management-agent-addon(d4e9d0db-3712-4daf-93e2-230aa9ba8e99)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" podUID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99"
Jul 23 00:12:31.781742 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:31.781723    2565 scope.go:117] "RemoveContainer" containerID="95d5660449b43f49c74b60d1f15b9ee082727c505bf2154d3c3ca81bb70b029a"
Jul 23 00:12:32.059711 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.059601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:32.059711 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.059666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:12:32.061910 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.061885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/21cfc68c-5751-42f7-a018-383e03338cd7-metrics-tls\") pod \"dns-default-r8cbw\" (UID: \"21cfc68c-5751-42f7-a018-383e03338cd7\") " pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:32.062136 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.062115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3101d750-8df5-4897-9428-b552c33a6c74-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5kjdk\" (UID: \"3101d750-8df5-4897-9428-b552c33a6c74\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:12:32.065388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.065355    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhhqg\""
Jul 23 00:12:32.065388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.065364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t88zl\""
Jul 23 00:12:32.073449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.073430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:32.073535 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.073480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"
Jul 23 00:12:32.205906 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.205876    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8cbw"]
Jul 23 00:12:32.208918 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:12:32.208888    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cfc68c_5751_42f7_a018_383e03338cd7.slice/crio-b6a208552672aa3a2e8b191dfbc77456b3f795c1b735bcc933e415b49f232e20 WatchSource:0}: Error finding container b6a208552672aa3a2e8b191dfbc77456b3f795c1b735bcc933e415b49f232e20: Status 404 returned error can't find the container with id b6a208552672aa3a2e8b191dfbc77456b3f795c1b735bcc933e415b49f232e20
Jul 23 00:12:32.221673 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.221624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk"]
Jul 23 00:12:32.223880 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:12:32.223853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3101d750_8df5_4897_9428_b552c33a6c74.slice/crio-568d2237df503ba53d2382287349ca2259de297ba5f7cb6a0acf7b3b72d76dce WatchSource:0}: Error finding container 568d2237df503ba53d2382287349ca2259de297ba5f7cb6a0acf7b3b72d76dce: Status 404 returned error can't find the container with id 568d2237df503ba53d2382287349ca2259de297ba5f7cb6a0acf7b3b72d76dce
Jul 23 00:12:32.339067 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.338990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:12:32.356104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.356071    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" podUID="561926cd-363d-4471-a8ea-f1a3878284aa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 00:12:32.369918 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.369896    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg"
Jul 23 00:12:32.426188 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.426163    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:12:32.779611 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.779583    2565 scope.go:117] "RemoveContainer" containerID="db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"
Jul 23 00:12:32.780064 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:32.779772    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-78895ddc9c-nschw_open-cluster-management-agent-addon(d4e9d0db-3712-4daf-93e2-230aa9ba8e99)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" podUID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99"
Jul 23 00:12:32.780330 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.780307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk" event={"ID":"3101d750-8df5-4897-9428-b552c33a6c74","Type":"ContainerStarted","Data":"568d2237df503ba53d2382287349ca2259de297ba5f7cb6a0acf7b3b72d76dce"}
Jul 23 00:12:32.781193 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.781173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8cbw" event={"ID":"21cfc68c-5751-42f7-a018-383e03338cd7","Type":"ContainerStarted","Data":"b6a208552672aa3a2e8b191dfbc77456b3f795c1b735bcc933e415b49f232e20"}
Jul 23 00:12:32.782477 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:32.782430    2565 scope.go:117] "RemoveContainer" containerID="68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba"
Jul 23 00:12:32.782651 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:32.782632    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-54546bc5c6-vcrpg_open-cluster-management-agent-addon(5d0a5061-c41e-46f4-8f71-1022202698b9)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" podUID="5d0a5061-c41e-46f4-8f71-1022202698b9"
Jul 23 00:12:33.786923 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:33.786891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk" event={"ID":"3101d750-8df5-4897-9428-b552c33a6c74","Type":"ContainerStarted","Data":"014062d6f5cc7bf9f4d096c7be3f4d18b959e1d0a7caf17e4f94b1666d3eae70"}
Jul 23 00:12:33.787374 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:33.787156    2565 scope.go:117] "RemoveContainer" containerID="db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"
Jul 23 00:12:33.787374 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:33.787294    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-78895ddc9c-nschw_open-cluster-management-agent-addon(d4e9d0db-3712-4daf-93e2-230aa9ba8e99)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" podUID="d4e9d0db-3712-4daf-93e2-230aa9ba8e99"
Jul 23 00:12:33.801797 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:33.801751    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5kjdk" podStartSLOduration=268.562431731 podStartE2EDuration="4m29.801737053s" totalImagesPullingTime="1.239305322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:33.800897018 +0000 UTC m=+288.654466702" watchObservedRunningTime="2026-07-23 00:12:33.801737053 +0000 UTC m=+288.655306704"
Jul 23 00:12:35.794213 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:35.794175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8cbw" event={"ID":"21cfc68c-5751-42f7-a018-383e03338cd7","Type":"ContainerStarted","Data":"2a8456e2a8b6937111cfc2bccc2f218ca814564e7d1cf480ffd086226860f57f"}
Jul 23 00:12:35.794676 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:35.794237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8cbw" event={"ID":"21cfc68c-5751-42f7-a018-383e03338cd7","Type":"ContainerStarted","Data":"f4d1d7cd8dc5645bbc0925ef67cf3dd6c47913e403b4561aba2912fefae604ff"}
Jul 23 00:12:35.794676 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:35.794305    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:35.811311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:35.811270    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r8cbw" podStartSLOduration=251.07335043 podStartE2EDuration="4m13.811257106s" totalImagesPullingTime="2.737906676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:35.80987343 +0000 UTC m=+290.663443126" watchObservedRunningTime="2026-07-23 00:12:35.811257106 +0000 UTC m=+290.664826778"
Jul 23 00:12:36.695378 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:36.695345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:36.697592 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:36.697561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/481ac944-2252-4ac5-ba03-4d641edf2a2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56xk7\" (UID: \"481ac944-2252-4ac5-ba03-4d641edf2a2a\") " pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:36.966905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:36.966814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56xk7"
Jul 23 00:12:37.090101 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:37.090072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56xk7"]
Jul 23 00:12:37.092271 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:12:37.092247    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ac944_2252_4ac5_ba03_4d641edf2a2a.slice/crio-04d3dde3b7ff835f4e9003f55b9cd3a3304b16c685a6810bcd972f49bae71127 WatchSource:0}: Error finding container 04d3dde3b7ff835f4e9003f55b9cd3a3304b16c685a6810bcd972f49bae71127: Status 404 returned error can't find the container with id 04d3dde3b7ff835f4e9003f55b9cd3a3304b16c685a6810bcd972f49bae71127
Jul 23 00:12:37.801914 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:37.801868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56xk7" event={"ID":"481ac944-2252-4ac5-ba03-4d641edf2a2a","Type":"ContainerStarted","Data":"04d3dde3b7ff835f4e9003f55b9cd3a3304b16c685a6810bcd972f49bae71127"}
Jul 23 00:12:42.355956 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:42.355920    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" podUID="561926cd-363d-4471-a8ea-f1a3878284aa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 00:12:42.720460 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:12:42.720371    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8vgc5" podUID="3ac5aa4f-7a25-4e1c-9f7a-579072177156"
Jul 23 00:12:42.822385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:42.822305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56xk7" event={"ID":"481ac944-2252-4ac5-ba03-4d641edf2a2a","Type":"ContainerStarted","Data":"75125c3815496dc866e7a7fa505e9524349d49cb0670de4d956b959251a48114"}
Jul 23 00:12:45.580920 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.580889    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 00:12:45.721214 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.721184    2565 scope.go:117] "RemoveContainer" containerID="68ed55539c4595c1bca29474c6c3b251b5356b86e48a277944e24a17569922ba"
Jul 23 00:12:45.721361 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.721292    2565 scope.go:117] "RemoveContainer" containerID="db2c8431a6daf379f3070ae4360bc2efeccdff9bf2ba15b132b4033ed8dfadbd"
Jul 23 00:12:45.799691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.799661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r8cbw"
Jul 23 00:12:45.834210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.834175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56xk7" event={"ID":"481ac944-2252-4ac5-ba03-4d641edf2a2a","Type":"ContainerStarted","Data":"359d0e6a62eef4e69a5187199d62484997def7649681f5de02ff6cbdfc32a6b9"}
Jul 23 00:12:45.835937 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.835907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54546bc5c6-vcrpg" event={"ID":"5d0a5061-c41e-46f4-8f71-1022202698b9","Type":"ContainerStarted","Data":"6836d4b088817e624e2a077f0b775ea65d8d97d4e3b12960c5841bef1b4f3b46"}
Jul 23 00:12:45.837626 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.837591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw" event={"ID":"d4e9d0db-3712-4daf-93e2-230aa9ba8e99","Type":"ContainerStarted","Data":"14faff8f9b850e45f6586dac0d7d7a55464c1c2fa80b3d15c5b4c8b89c70cb03"}
Jul 23 00:12:45.837891 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.837876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:12:45.839201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:45.839181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78895ddc9c-nschw"
Jul 23 00:12:49.853467 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:49.853431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56xk7" event={"ID":"481ac944-2252-4ac5-ba03-4d641edf2a2a","Type":"ContainerStarted","Data":"d21060b5e0160740300e8076a9321fa9574620435a8748216e25e5cbc7a5ad2b"}
Jul 23 00:12:49.873492 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:49.873439    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-56xk7" podStartSLOduration=34.092099967 podStartE2EDuration="45.87342374s" totalImagesPullingTime="11.781323773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:12:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:12:49.872590534 +0000 UTC m=+304.726160208" watchObservedRunningTime="2026-07-23 00:12:49.87342374 +0000 UTC m=+304.726993413"
Jul 23 00:12:52.356018 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.355964    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" podUID="561926cd-363d-4471-a8ea-f1a3878284aa" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 00:12:52.356390 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.356045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x"
Jul 23 00:12:52.356547 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.356516    2565 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b89ce564600ee830f57d332ca60c6e200f64c0d042edf5cf8926fc2f395a8d90"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 23 00:12:52.356598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.356567    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" podUID="561926cd-363d-4471-a8ea-f1a3878284aa" containerName="service-proxy" containerID="cri-o://b89ce564600ee830f57d332ca60c6e200f64c0d042edf5cf8926fc2f395a8d90" gracePeriod=30
Jul 23 00:12:52.865096 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.865061    2565 generic.go:356] "Generic (PLEG): container finished" podID="561926cd-363d-4471-a8ea-f1a3878284aa" containerID="b89ce564600ee830f57d332ca60c6e200f64c0d042edf5cf8926fc2f395a8d90" exitCode=2
Jul 23 00:12:52.865262 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.865126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerDied","Data":"b89ce564600ee830f57d332ca60c6e200f64c0d042edf5cf8926fc2f395a8d90"}
Jul 23 00:12:52.865262 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:52.865172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f9598457-bbh8x" event={"ID":"561926cd-363d-4471-a8ea-f1a3878284aa","Type":"ContainerStarted","Data":"b173baef171b32deb55859ecc5461fdada466dee9d138ff696d51a485bf1df74"}
Jul 23 00:12:54.720252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:12:54.720203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:13:01.482576 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.482546    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7cxgz"]
Jul 23 00:13:01.485747 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.485726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.488471 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.488446    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 00:13:01.489914 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.489371    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 00:13:01.489914 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.489773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 00:13:01.490098 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.490055    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrwh\""
Jul 23 00:13:01.490184 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.490162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 00:13:01.490318 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.490297    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 00:13:01.490518 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.490395    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 00:13:01.674154 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-root\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674154 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-textfile\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7zv\" (UniqueName: \"kubernetes.io/projected/ad736d8f-ce73-4e1e-843f-377295ac59f1-kube-api-access-dq7zv\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-metrics-client-ca\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-sys\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-wtmp\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.674610 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.674417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775695 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-root\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-textfile\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7zv\" (UniqueName: \"kubernetes.io/projected/ad736d8f-ce73-4e1e-843f-377295ac59f1-kube-api-access-dq7zv\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-metrics-client-ca\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-root\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-sys\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-sys\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.775885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.776389 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.775897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-wtmp\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.776389 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.776044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-wtmp\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.776389 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:13:01.776080    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 23 00:13:01.776389 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:13:01.776154    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls podName:ad736d8f-ce73-4e1e-843f-377295ac59f1 nodeName:}" failed. No retries permitted until 2026-07-23 00:13:02.276138362 +0000 UTC m=+317.129708014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls") pod "node-exporter-7cxgz" (UID: "ad736d8f-ce73-4e1e-843f-377295ac59f1") : secret "node-exporter-tls" not found
Jul 23 00:13:01.776389 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.776279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-textfile\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.776553 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.776461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.776553 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.776470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad736d8f-ce73-4e1e-843f-377295ac59f1-metrics-client-ca\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.777980 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.777950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:01.784180 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:01.784160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7zv\" (UniqueName: \"kubernetes.io/projected/ad736d8f-ce73-4e1e-843f-377295ac59f1-kube-api-access-dq7zv\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:02.280260 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:02.280215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:02.282507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:02.282478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ad736d8f-ce73-4e1e-843f-377295ac59f1-node-exporter-tls\") pod \"node-exporter-7cxgz\" (UID: \"ad736d8f-ce73-4e1e-843f-377295ac59f1\") " pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:02.396656 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:02.396626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cxgz"
Jul 23 00:13:02.409452 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:13:02.409426    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad736d8f_ce73_4e1e_843f_377295ac59f1.slice/crio-78b907235d0de00dc81ce836789e6524538b2dd898456b17d344e23d72f08606 WatchSource:0}: Error finding container 78b907235d0de00dc81ce836789e6524538b2dd898456b17d344e23d72f08606: Status 404 returned error can't find the container with id 78b907235d0de00dc81ce836789e6524538b2dd898456b17d344e23d72f08606
Jul 23 00:13:02.411213 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:02.411192    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:13:02.897299 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:02.897261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cxgz" event={"ID":"ad736d8f-ce73-4e1e-843f-377295ac59f1","Type":"ContainerStarted","Data":"78b907235d0de00dc81ce836789e6524538b2dd898456b17d344e23d72f08606"}
Jul 23 00:13:03.903030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:03.902927    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad736d8f-ce73-4e1e-843f-377295ac59f1" containerID="b8f8ecd666d6f57dea6947e2520f431997cb43f919bd8bb9528b494d647896f1" exitCode=0
Jul 23 00:13:03.903030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:03.903011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cxgz" event={"ID":"ad736d8f-ce73-4e1e-843f-377295ac59f1","Type":"ContainerDied","Data":"b8f8ecd666d6f57dea6947e2520f431997cb43f919bd8bb9528b494d647896f1"}
Jul 23 00:13:04.908394 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:04.908358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cxgz" event={"ID":"ad736d8f-ce73-4e1e-843f-377295ac59f1","Type":"ContainerStarted","Data":"0054af154a96beaa81d5f06490f76b2d04cea5f3e4998a87f3bb89639f36b9bd"}
Jul 23 00:13:04.908864 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:04.908408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cxgz" event={"ID":"ad736d8f-ce73-4e1e-843f-377295ac59f1","Type":"ContainerStarted","Data":"0a95475e2b70476d3701bdf160d2fb52d9d801048ebf729d54f3e93ccfd87039"}
Jul 23 00:13:04.928003 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:04.927941    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7cxgz" podStartSLOduration=2.7831160390000003 podStartE2EDuration="3.927925093s" totalImagesPullingTime="1.082929409s" totalInitContainerRuntime="61.879645ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:13:04.926694643 +0000 UTC m=+319.780264317" watchObservedRunningTime="2026-07-23 00:13:04.927925093 +0000 UTC m=+319.781494765"
Jul 23 00:13:59.708269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:59.708232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:13:59.710491 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:59.710469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac5aa4f-7a25-4e1c-9f7a-579072177156-metrics-certs\") pod \"network-metrics-daemon-8vgc5\" (UID: \"3ac5aa4f-7a25-4e1c-9f7a-579072177156\") " pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:13:59.824256 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:59.824227    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pff25\""
Jul 23 00:13:59.831850 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:59.831818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8vgc5"
Jul 23 00:13:59.951508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:13:59.951440    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8vgc5"]
Jul 23 00:13:59.954134 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:13:59.954106    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac5aa4f_7a25_4e1c_9f7a_579072177156.slice/crio-48581abfbd276553186c6720983050d71b1d93d131fd6300dfc3ad7058061bd7 WatchSource:0}: Error finding container 48581abfbd276553186c6720983050d71b1d93d131fd6300dfc3ad7058061bd7: Status 404 returned error can't find the container with id 48581abfbd276553186c6720983050d71b1d93d131fd6300dfc3ad7058061bd7
Jul 23 00:14:00.086701 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:14:00.086664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vgc5" event={"ID":"3ac5aa4f-7a25-4e1c-9f7a-579072177156","Type":"ContainerStarted","Data":"48581abfbd276553186c6720983050d71b1d93d131fd6300dfc3ad7058061bd7"}
Jul 23 00:14:01.090788 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:14:01.090739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vgc5" event={"ID":"3ac5aa4f-7a25-4e1c-9f7a-579072177156","Type":"ContainerStarted","Data":"b090445a8afa59a2a49bc6ac4f1aaac86fb365621865f07666bdb70ac132eec4"}
Jul 23 00:14:02.095880 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:14:02.095839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8vgc5" event={"ID":"3ac5aa4f-7a25-4e1c-9f7a-579072177156","Type":"ContainerStarted","Data":"ef8fcf415fb4b021433a65644cd6dd1406973994d32911601e87765bf8627752"}
Jul 23 00:16:56.531099 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.531040    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8vgc5" podStartSLOduration=549.486187553 podStartE2EDuration="9m10.531018486s" totalImagesPullingTime="1.044830933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:14:02.114857316 +0000 UTC m=+376.968426989" watchObservedRunningTime="2026-07-23 00:16:56.531018486 +0000 UTC m=+551.384588167"
Jul 23 00:16:56.532318 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.532286    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"]
Jul 23 00:16:56.536230 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.536203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.539371 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.539347    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 23 00:16:56.539487 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.539467    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 23 00:16:56.540526 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.540509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rg5v4\""
Jul 23 00:16:56.543020 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.542996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"]
Jul 23 00:16:56.613218 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.613177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm5j\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-kube-api-access-bqm5j\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.613386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.613248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.713522 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.713489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.713685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.713538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm5j\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-kube-api-access-bqm5j\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.722509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.722484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.722639 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.722530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm5j\" (UniqueName: \"kubernetes.io/projected/e42bb981-9f51-47b4-829b-392dbc636109-kube-api-access-bqm5j\") pod \"cert-manager-cainjector-559cdd8c87-l7j49\" (UID: \"e42bb981-9f51-47b4-829b-392dbc636109\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.846353 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.846274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"
Jul 23 00:16:56.966749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:56.966645    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-l7j49"]
Jul 23 00:16:56.969340 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:16:56.969314    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode42bb981_9f51_47b4_829b_392dbc636109.slice/crio-59988b5389d907b06a276b07c8f5e35097ca0c747d2282acf78587536df80a9d WatchSource:0}: Error finding container 59988b5389d907b06a276b07c8f5e35097ca0c747d2282acf78587536df80a9d: Status 404 returned error can't find the container with id 59988b5389d907b06a276b07c8f5e35097ca0c747d2282acf78587536df80a9d
Jul 23 00:16:57.657566 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:16:57.657528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49" event={"ID":"e42bb981-9f51-47b4-829b-392dbc636109","Type":"ContainerStarted","Data":"59988b5389d907b06a276b07c8f5e35097ca0c747d2282acf78587536df80a9d"}
Jul 23 00:17:00.670360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:00.670323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49" event={"ID":"e42bb981-9f51-47b4-829b-392dbc636109","Type":"ContainerStarted","Data":"b8d5e4e32fa658df6a67c37c155f51da97c114725702fb32f91ad4bf5aa200d4"}
Jul 23 00:17:00.685763 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:00.685719    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-l7j49" podStartSLOduration=1.932904028 podStartE2EDuration="4.685704433s" totalImagesPullingTime="2.752800405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:00.685338759 +0000 UTC m=+555.538908431" watchObservedRunningTime="2026-07-23 00:17:00.685704433 +0000 UTC m=+555.539274105"
Jul 23 00:17:27.379077 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.379042    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"]
Jul 23 00:17:27.383601 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.383579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.387642 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.387618    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mblxv\""
Jul 23 00:17:27.387752 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.387659    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 00:17:27.387752 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.387698    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 00:17:27.387752 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.387706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 00:17:27.387911 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.387788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 00:17:27.404039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.404016    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"]
Jul 23 00:17:27.524133 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.524096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.524395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.524141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.524395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.524168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfqb6\" (UniqueName: \"kubernetes.io/projected/cb6c2601-95ba-4c7d-a712-067396648a9d-kube-api-access-nfqb6\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.524395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.524196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.524395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.524222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.624987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.625043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.625059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfqb6\" (UniqueName: \"kubernetes.io/projected/cb6c2601-95ba-4c7d-a712-067396648a9d-kube-api-access-nfqb6\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.625080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.625105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.625453 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.625382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-manifests\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.627240 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.627214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb6c2601-95ba-4c7d-a712-067396648a9d-tmp\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.627444 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.627425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.627489 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.627454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cb6c2601-95ba-4c7d-a712-067396648a9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.633111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.633053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfqb6\" (UniqueName: \"kubernetes.io/projected/cb6c2601-95ba-4c7d-a712-067396648a9d-kube-api-access-nfqb6\") pod \"opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk\" (UID: \"cb6c2601-95ba-4c7d-a712-067396648a9d\") " pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.692922 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.692887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:27.843073 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:27.843049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"]
Jul 23 00:17:27.845439 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:27.845419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6c2601_95ba_4c7d_a712_067396648a9d.slice/crio-fe92b67f6f4735f1832ec69ed586dde8ee76c7f42d994f31844093d1ace0b464 WatchSource:0}: Error finding container fe92b67f6f4735f1832ec69ed586dde8ee76c7f42d994f31844093d1ace0b464: Status 404 returned error can't find the container with id fe92b67f6f4735f1832ec69ed586dde8ee76c7f42d994f31844093d1ace0b464
Jul 23 00:17:28.769194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:28.769157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk" event={"ID":"cb6c2601-95ba-4c7d-a712-067396648a9d","Type":"ContainerStarted","Data":"fe92b67f6f4735f1832ec69ed586dde8ee76c7f42d994f31844093d1ace0b464"}
Jul 23 00:17:31.781308 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:31.781275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk" event={"ID":"cb6c2601-95ba-4c7d-a712-067396648a9d","Type":"ContainerStarted","Data":"9b28c60d6072af8351f982b49c63453ddc787b2f64571c978a5049f1cb9b7193"}
Jul 23 00:17:32.786115 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:32.786079    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb6c2601-95ba-4c7d-a712-067396648a9d" containerID="9b28c60d6072af8351f982b49c63453ddc787b2f64571c978a5049f1cb9b7193" exitCode=0
Jul 23 00:17:32.786511 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:32.786131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk" event={"ID":"cb6c2601-95ba-4c7d-a712-067396648a9d","Type":"ContainerDied","Data":"9b28c60d6072af8351f982b49c63453ddc787b2f64571c978a5049f1cb9b7193"}
Jul 23 00:17:33.791497 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:33.791465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk" event={"ID":"cb6c2601-95ba-4c7d-a712-067396648a9d","Type":"ContainerStarted","Data":"596219b1cf9e8f42afe6a75e1a3f148be5058fec30840ae8674a3711472cc3b5"}
Jul 23 00:17:33.791876 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:33.791577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:33.823794 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:33.823746    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk" podStartSLOduration=2.37878472 podStartE2EDuration="6.823732434s" totalImagesPullingTime="3.417289553s" totalInitContainerRuntime="1.027658161s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:33.821802493 +0000 UTC m=+588.675372165" watchObservedRunningTime="2026-07-23 00:17:33.823732434 +0000 UTC m=+588.677302106"
Jul 23 00:17:44.797374 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:44.797344    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk"
Jul 23 00:17:47.203820 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.203784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"]
Jul 23 00:17:47.206712 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.206692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.211163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.211139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 00:17:47.211290 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.211142    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 00:17:47.211290 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.211189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-62c26\""
Jul 23 00:17:47.211290 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.211191    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 00:17:47.211290 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.211236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 00:17:47.222315 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.222291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"]
Jul 23 00:17:47.258791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.258763    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq54c"]
Jul 23 00:17:47.260733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.260711    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.264259 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.264240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 23 00:17:47.264532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.264518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9mgfd\""
Jul 23 00:17:47.282163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.282139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq54c"]
Jul 23 00:17:47.363980 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.363937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d287c6-23be-4296-bf56-f8ef83a3272b-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.364155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.364000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d287c6-23be-4296-bf56-f8ef83a3272b-tmp\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.364155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.364032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.364155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.364102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vmv\" (UniqueName: \"kubernetes.io/projected/91d287c6-23be-4296-bf56-f8ef83a3272b-kube-api-access-d8vmv\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.364155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.364151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh75g\" (UniqueName: \"kubernetes.io/projected/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-kube-api-access-sh75g\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.465302 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.465222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vmv\" (UniqueName: \"kubernetes.io/projected/91d287c6-23be-4296-bf56-f8ef83a3272b-kube-api-access-d8vmv\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.465302 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.465265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh75g\" (UniqueName: \"kubernetes.io/projected/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-kube-api-access-sh75g\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.465302 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.465289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d287c6-23be-4296-bf56-f8ef83a3272b-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.465540 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.465413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d287c6-23be-4296-bf56-f8ef83a3272b-tmp\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.465540 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.465461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.465654 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:47.465562    2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 23 00:17:47.465654 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:47.465622    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert podName:1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3 nodeName:}" failed. No retries permitted until 2026-07-23 00:17:47.965607256 +0000 UTC m=+602.819176907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert") pod "odh-model-controller-7998c67dc4-mq54c" (UID: "1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3") : secret "odh-model-controller-webhook-cert" not found
Jul 23 00:17:47.467577 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.467548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91d287c6-23be-4296-bf56-f8ef83a3272b-tmp\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.467706 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.467689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d287c6-23be-4296-bf56-f8ef83a3272b-tls-certs\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.476279 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.476256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh75g\" (UniqueName: \"kubernetes.io/projected/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-kube-api-access-sh75g\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.476541 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.476523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vmv\" (UniqueName: \"kubernetes.io/projected/91d287c6-23be-4296-bf56-f8ef83a3272b-kube-api-access-d8vmv\") pod \"kube-auth-proxy-7447dc7bcf-hn8hx\" (UID: \"91d287c6-23be-4296-bf56-f8ef83a3272b\") " pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.519984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.519942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"
Jul 23 00:17:47.643536 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.643512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx"]
Jul 23 00:17:47.645828 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:47.645803    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d287c6_23be_4296_bf56_f8ef83a3272b.slice/crio-7a11be7dfac354601b0072de2655c370fdb99fdff814e8f2d1aab22c5190a737 WatchSource:0}: Error finding container 7a11be7dfac354601b0072de2655c370fdb99fdff814e8f2d1aab22c5190a737: Status 404 returned error can't find the container with id 7a11be7dfac354601b0072de2655c370fdb99fdff814e8f2d1aab22c5190a737
Jul 23 00:17:47.839167 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.839133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx" event={"ID":"91d287c6-23be-4296-bf56-f8ef83a3272b","Type":"ContainerStarted","Data":"7a11be7dfac354601b0072de2655c370fdb99fdff814e8f2d1aab22c5190a737"}
Jul 23 00:17:47.969247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.969214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:47.971629 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:47.971596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3-cert\") pod \"odh-model-controller-7998c67dc4-mq54c\" (UID: \"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3\") " pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:48.169931 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.169846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:48.240517 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.240484    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2478h"]
Jul 23 00:17:48.242662 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.242643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.246360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.246333    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sbcnv\""
Jul 23 00:17:48.246360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.246352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 23 00:17:48.258457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.258424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2478h"]
Jul 23 00:17:48.299717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.299696    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mq54c"]
Jul 23 00:17:48.302063 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:48.302041    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5eee87_e866_453d_ae6f_fe8dfa1d6ef3.slice/crio-52c6aeeffbf2dcd4270eb637b2e547d486611ce45c04a87b00b837f27627bfea WatchSource:0}: Error finding container 52c6aeeffbf2dcd4270eb637b2e547d486611ce45c04a87b00b837f27627bfea: Status 404 returned error can't find the container with id 52c6aeeffbf2dcd4270eb637b2e547d486611ce45c04a87b00b837f27627bfea
Jul 23 00:17:48.371330 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.371297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db779\" (UniqueName: \"kubernetes.io/projected/d6e13d01-01ee-4585-81bd-e88c40221074-kube-api-access-db779\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.371455 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.371384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.472633 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.472548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.472783 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.472638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db779\" (UniqueName: \"kubernetes.io/projected/d6e13d01-01ee-4585-81bd-e88c40221074-kube-api-access-db779\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.472783 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:48.472678    2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 23 00:17:48.472783 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:48.472753    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs podName:d6e13d01-01ee-4585-81bd-e88c40221074 nodeName:}" failed. No retries permitted until 2026-07-23 00:17:48.972733987 +0000 UTC m=+603.826303654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs") pod "model-serving-api-54d89bd5fc-2478h" (UID: "d6e13d01-01ee-4585-81bd-e88c40221074") : secret "model-serving-api-tls" not found
Jul 23 00:17:48.485905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.485878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db779\" (UniqueName: \"kubernetes.io/projected/d6e13d01-01ee-4585-81bd-e88c40221074-kube-api-access-db779\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.509105 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.509066    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:17:48.511205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.511177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.514104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.514065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 00:17:48.514104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.514082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5hjrl\""
Jul 23 00:17:48.514295 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.514083    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 00:17:48.514295 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.514074    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 00:17:48.514418 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.514403    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 00:17:48.525987 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.525951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:17:48.674837 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.674787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.674853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.674906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.674941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgmr\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675178 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.675029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675178 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.675079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.675178 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.675134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776204 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgmr\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776565 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776565 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.776979 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.776951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.778905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.778874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.779015 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.778902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.779095 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.779023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.779240 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.779220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.784384 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.784358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgmr\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.784491 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.784365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token\") pod \"istiod-openshift-gateway-7574459656-2wwdt\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.822921 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.822893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:48.844077 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.844047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c" event={"ID":"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3","Type":"ContainerStarted","Data":"52c6aeeffbf2dcd4270eb637b2e547d486611ce45c04a87b00b837f27627bfea"}
Jul 23 00:17:48.956999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.956900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:17:48.959041 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:48.959014    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9f673b_bc5b_49a7_ba24_48d7f7522ed9.slice/crio-127fffaac58f4dbb46c9ad6bccd69ff88a8d78c656fa8f0633c1e15ea47c3f6e WatchSource:0}: Error finding container 127fffaac58f4dbb46c9ad6bccd69ff88a8d78c656fa8f0633c1e15ea47c3f6e: Status 404 returned error can't find the container with id 127fffaac58f4dbb46c9ad6bccd69ff88a8d78c656fa8f0633c1e15ea47c3f6e
Jul 23 00:17:48.978369 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.978293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:48.989943 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:48.980567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e13d01-01ee-4585-81bd-e88c40221074-tls-certs\") pod \"model-serving-api-54d89bd5fc-2478h\" (UID: \"d6e13d01-01ee-4585-81bd-e88c40221074\") " pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:49.154033 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:49.153947    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:49.286209 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:49.286174    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2478h"]
Jul 23 00:17:49.289639 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:49.289609    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e13d01_01ee_4585_81bd_e88c40221074.slice/crio-f680f70a818611e4ac2914f28c2fa208d412cf0f8a1abc8e50e84770c23d4e51 WatchSource:0}: Error finding container f680f70a818611e4ac2914f28c2fa208d412cf0f8a1abc8e50e84770c23d4e51: Status 404 returned error can't find the container with id f680f70a818611e4ac2914f28c2fa208d412cf0f8a1abc8e50e84770c23d4e51
Jul 23 00:17:49.850612 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:49.850571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2478h" event={"ID":"d6e13d01-01ee-4585-81bd-e88c40221074","Type":"ContainerStarted","Data":"f680f70a818611e4ac2914f28c2fa208d412cf0f8a1abc8e50e84770c23d4e51"}
Jul 23 00:17:49.852057 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:49.852022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" event={"ID":"be9f673b-bc5b-49a7-ba24-48d7f7522ed9","Type":"ContainerStarted","Data":"127fffaac58f4dbb46c9ad6bccd69ff88a8d78c656fa8f0633c1e15ea47c3f6e"}
Jul 23 00:17:53.376007 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.375956    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:17:53.376316 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.376037    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:17:53.873503 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.873454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx" event={"ID":"91d287c6-23be-4296-bf56-f8ef83a3272b","Type":"ContainerStarted","Data":"ce9722f961761c2c04120d5468d0642238a6bdd4be6cd61ec44494cac68e9515"}
Jul 23 00:17:53.875599 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.875547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c" event={"ID":"1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3","Type":"ContainerStarted","Data":"13e9165434a8d4a92ea6e28f02cbab1568a8a697d55194895088a09771ffdfc0"}
Jul 23 00:17:53.878404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.878374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2478h" event={"ID":"d6e13d01-01ee-4585-81bd-e88c40221074","Type":"ContainerStarted","Data":"9cc90ed16e2a6b0b6cc52d66336c6687a026ab93bf2f359937e2af169d90ff17"}
Jul 23 00:17:53.878911 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.878886    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:17:53.881041 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.881015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" event={"ID":"be9f673b-bc5b-49a7-ba24-48d7f7522ed9","Type":"ContainerStarted","Data":"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"}
Jul 23 00:17:53.881556 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.881532    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:53.881745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.881721    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:17:53.895734 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.895681    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7447dc7bcf-hn8hx" podStartSLOduration=1.174684334 podStartE2EDuration="6.895661631s" totalImagesPullingTime="5.720977297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:53.894293755 +0000 UTC m=+608.747863439" watchObservedRunningTime="2026-07-23 00:17:53.895661631 +0000 UTC m=+608.749231303"
Jul 23 00:17:53.919717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.919664    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2478h" podStartSLOduration=1.831929638 podStartE2EDuration="5.919647707s" totalImagesPullingTime="4.087718069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:53.919269751 +0000 UTC m=+608.772839426" watchObservedRunningTime="2026-07-23 00:17:53.919647707 +0000 UTC m=+608.773217380"
Jul 23 00:17:53.946896 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.946844    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c" podStartSLOduration=1.8654179819999999 podStartE2EDuration="6.94682619s" totalImagesPullingTime="5.081408208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:53.946397692 +0000 UTC m=+608.799967366" watchObservedRunningTime="2026-07-23 00:17:53.94682619 +0000 UTC m=+608.800395915"
Jul 23 00:17:53.969329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:53.969273    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" podStartSLOduration=1.5544857520000002 podStartE2EDuration="5.969252972s" totalImagesPullingTime="4.41476722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:17:53.967053301 +0000 UTC m=+608.820622991" watchObservedRunningTime="2026-07-23 00:17:53.969252972 +0000 UTC m=+608.822822647"
Jul 23 00:17:54.886660 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:54.886633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:17:57.527812 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.527781    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"]
Jul 23 00:17:57.531061 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.531039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.533746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.533722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 23 00:17:57.533746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.533739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l96fr\""
Jul 23 00:17:57.541715 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.541689    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"]
Jul 23 00:17:57.544753 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.544735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"]
Jul 23 00:17:57.544834 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.544813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:57.548633 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.548615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 23 00:17:57.548733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.548662    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v5578\""
Jul 23 00:17:57.554278 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.554248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqf7h\" (UniqueName: \"kubernetes.io/projected/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-kube-api-access-tqf7h\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.554379 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.554345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.565429 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.565408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"]
Jul 23 00:17:57.655098 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.655052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8jd\" (UniqueName: \"kubernetes.io/projected/c64a6008-4724-4601-a990-30b4f28eac42-kube-api-access-hf8jd\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:57.655301 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.655163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.655301 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.655208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqf7h\" (UniqueName: \"kubernetes.io/projected/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-kube-api-access-tqf7h\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.655301 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.655248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:57.655460 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:57.655294    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 23 00:17:57.655460 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:57.655357    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert podName:a6e29054-2d36-4542-9fb1-8dc19acc3f5b nodeName:}" failed. No retries permitted until 2026-07-23 00:17:58.155337678 +0000 UTC m=+613.008907332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert") pod "kserve-controller-manager-d6f4b65b8-hdq46" (UID: "a6e29054-2d36-4542-9fb1-8dc19acc3f5b") : secret "kserve-webhook-server-cert" not found
Jul 23 00:17:57.668814 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.668790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqf7h\" (UniqueName: \"kubernetes.io/projected/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-kube-api-access-tqf7h\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:57.756312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.756277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:57.756490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.756333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8jd\" (UniqueName: \"kubernetes.io/projected/c64a6008-4724-4601-a990-30b4f28eac42-kube-api-access-hf8jd\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:57.756490 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:57.756429    2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 23 00:17:57.756601 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:17:57.756499    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert podName:c64a6008-4724-4601-a990-30b4f28eac42 nodeName:}" failed. No retries permitted until 2026-07-23 00:17:58.256479235 +0000 UTC m=+613.110048900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert") pod "llmisvc-controller-manager-777c6bd7d5-ksgsr" (UID: "c64a6008-4724-4601-a990-30b4f28eac42") : secret "llmisvc-webhook-server-cert" not found
Jul 23 00:17:57.773393 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:57.773366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8jd\" (UniqueName: \"kubernetes.io/projected/c64a6008-4724-4601-a990-30b4f28eac42-kube-api-access-hf8jd\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:58.159642 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.159611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:58.162023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.162004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e29054-2d36-4542-9fb1-8dc19acc3f5b-cert\") pod \"kserve-controller-manager-d6f4b65b8-hdq46\" (UID: \"a6e29054-2d36-4542-9fb1-8dc19acc3f5b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:58.260808 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.260771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:58.263323 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.263300    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c64a6008-4724-4601-a990-30b4f28eac42-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ksgsr\" (UID: \"c64a6008-4724-4601-a990-30b4f28eac42\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:58.440880 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.440799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:17:58.455982 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.455941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:17:58.610023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.609996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"]
Jul 23 00:17:58.611480 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:58.611456    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e29054_2d36_4542_9fb1_8dc19acc3f5b.slice/crio-d9ad1bf6feda407675c03b474843677c5889324b6bb61b434db58f70caada79e WatchSource:0}: Error finding container d9ad1bf6feda407675c03b474843677c5889324b6bb61b434db58f70caada79e: Status 404 returned error can't find the container with id d9ad1bf6feda407675c03b474843677c5889324b6bb61b434db58f70caada79e
Jul 23 00:17:58.631487 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.631465    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"]
Jul 23 00:17:58.633238 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:17:58.633213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc64a6008_4724_4601_a990_30b4f28eac42.slice/crio-dbd21c1df1dfc7b507104e20879ff94508e6d673932530afc67ba305b6824f3a WatchSource:0}: Error finding container dbd21c1df1dfc7b507104e20879ff94508e6d673932530afc67ba305b6824f3a: Status 404 returned error can't find the container with id dbd21c1df1dfc7b507104e20879ff94508e6d673932530afc67ba305b6824f3a
Jul 23 00:17:58.902582 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.902547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr" event={"ID":"c64a6008-4724-4601-a990-30b4f28eac42","Type":"ContainerStarted","Data":"dbd21c1df1dfc7b507104e20879ff94508e6d673932530afc67ba305b6824f3a"}
Jul 23 00:17:58.903784 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:17:58.903753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46" event={"ID":"a6e29054-2d36-4542-9fb1-8dc19acc3f5b","Type":"ContainerStarted","Data":"d9ad1bf6feda407675c03b474843677c5889324b6bb61b434db58f70caada79e"}
Jul 23 00:18:00.325327 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.322791    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:18:00.325327 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.323132    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" podUID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" containerName="discovery" containerID="cri-o://c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640" gracePeriod=30
Jul 23 00:18:00.587752 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.587680    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:18:00.678000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.677943    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdgmr\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678088    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.678847 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.678801    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:18:00.681153 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.681117    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token" (OuterVolumeSpecName: "istio-token") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:18:00.681271 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.681157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:18:00.681334 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.681280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr" (OuterVolumeSpecName: "kube-api-access-xdgmr") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "kube-api-access-xdgmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:18:00.681395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.681367    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:18:00.681451 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.681431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts" (OuterVolumeSpecName: "cacerts") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:18:00.778775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs\") pod \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\" (UID: \"be9f673b-bc5b-49a7-ba24-48d7f7522ed9\") "
Jul 23 00:18:00.778984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778921    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-cacerts\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.778984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778939    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-dns-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.778984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778951    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-kubeconfig\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.778984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778963    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-token\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.779197 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.778991    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgmr\" (UniqueName: \"kubernetes.io/projected/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-kube-api-access-xdgmr\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.779197 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.779004    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-istio-csr-ca-configmap\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.781041 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.781008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs" (OuterVolumeSpecName: "local-certs") pod "be9f673b-bc5b-49a7-ba24-48d7f7522ed9" (UID: "be9f673b-bc5b-49a7-ba24-48d7f7522ed9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:18:00.879936 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.879842    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9f673b-bc5b-49a7-ba24-48d7f7522ed9-local-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:00.914803 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.914761    2565 generic.go:356] "Generic (PLEG): container finished" podID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" containerID="c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640" exitCode=0
Jul 23 00:18:00.915064 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.914874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" event={"ID":"be9f673b-bc5b-49a7-ba24-48d7f7522ed9","Type":"ContainerDied","Data":"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"}
Jul 23 00:18:00.915064 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.914924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt" event={"ID":"be9f673b-bc5b-49a7-ba24-48d7f7522ed9","Type":"ContainerDied","Data":"127fffaac58f4dbb46c9ad6bccd69ff88a8d78c656fa8f0633c1e15ea47c3f6e"}
Jul 23 00:18:00.915064 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.914951    2565 scope.go:117] "RemoveContainer" containerID="c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"
Jul 23 00:18:00.915234 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.915143    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"
Jul 23 00:18:00.960517 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.960479    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:18:00.972000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:00.971957    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2wwdt"]
Jul 23 00:18:01.345206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.345022    2565 scope.go:117] "RemoveContainer" containerID="c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"
Jul 23 00:18:01.345481 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:18:01.345391    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640\": container with ID starting with c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640 not found: ID does not exist" containerID="c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"
Jul 23 00:18:01.345481 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.345425    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640"} err="failed to get container status \"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640\": rpc error: code = NotFound desc = could not find container \"c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640\": container with ID starting with c10be44ed966989c141f4588b0a4d4640d613b56ac903cc307756df1d001d640 not found: ID does not exist"
Jul 23 00:18:01.726340 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.726262    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" path="/var/lib/kubelet/pods/be9f673b-bc5b-49a7-ba24-48d7f7522ed9/volumes"
Jul 23 00:18:01.920688 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.920649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46" event={"ID":"a6e29054-2d36-4542-9fb1-8dc19acc3f5b","Type":"ContainerStarted","Data":"f2d5873d40dc2586744444d1054c59a921fec9223e1097c6f89ee7f320e9c42f"}
Jul 23 00:18:01.920878 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.920768    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:18:01.944284 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:01.944225    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46" podStartSLOduration=2.1643284720000002 podStartE2EDuration="4.944205105s" totalImagesPullingTime="2.779876633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:18:01.94190526 +0000 UTC m=+616.795474935" watchObservedRunningTime="2026-07-23 00:18:01.944205105 +0000 UTC m=+616.797774780"
Jul 23 00:18:02.927256 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:02.927213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr" event={"ID":"c64a6008-4724-4601-a990-30b4f28eac42","Type":"ContainerStarted","Data":"ab3913d48b15f186f0bbc6b7cd561d9b8486a56ce8ddecceda657dc6f27c29bb"}
Jul 23 00:18:02.927661 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:02.927321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:18:02.943112 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:02.943061    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr" podStartSLOduration=2.253063326 podStartE2EDuration="5.943042865s" totalImagesPullingTime="3.689979539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:18:02.942761177 +0000 UTC m=+617.796330850" watchObservedRunningTime="2026-07-23 00:18:02.943042865 +0000 UTC m=+617.796612538"
Jul 23 00:18:04.887468 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:04.887437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mq54c"
Jul 23 00:18:05.893624 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:05.893595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2478h"
Jul 23 00:18:32.933144 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:32.933115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hdq46"
Jul 23 00:18:33.933766 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:33.933738    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ksgsr"
Jul 23 00:18:48.115429 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.115389    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:48.115897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.115780    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" containerName="discovery"
Jul 23 00:18:48.115897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.115798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" containerName="discovery"
Jul 23 00:18:48.115897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.115872    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9f673b-bc5b-49a7-ba24-48d7f7522ed9" containerName="discovery"
Jul 23 00:18:48.118317 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.118296    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:48.121296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.121278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-tmzrb\""
Jul 23 00:18:48.121440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.121277    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 00:18:48.121440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.121301    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 00:18:48.130128 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.130097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:48.216402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.216371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl66p\" (UniqueName: \"kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p\") pod \"kuadrant-operator-catalog-mltc9\" (UID: \"2a3fd8ee-346a-4805-bedf-d63d963aa40c\") " pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:48.316876 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.316835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl66p\" (UniqueName: \"kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p\") pod \"kuadrant-operator-catalog-mltc9\" (UID: \"2a3fd8ee-346a-4805-bedf-d63d963aa40c\") " pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:48.325203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.325172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl66p\" (UniqueName: \"kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p\") pod \"kuadrant-operator-catalog-mltc9\" (UID: \"2a3fd8ee-346a-4805-bedf-d63d963aa40c\") " pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:48.428109 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.428013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:48.480134 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.480102    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:48.565871 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.565839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:48.568074 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:18:48.568044    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3fd8ee_346a_4805_bedf_d63d963aa40c.slice/crio-ffdf2d153d38110ba1b07f1421157c4db916e2523c715ef58d7a9e3784385576 WatchSource:0}: Error finding container ffdf2d153d38110ba1b07f1421157c4db916e2523c715ef58d7a9e3784385576: Status 404 returned error can't find the container with id ffdf2d153d38110ba1b07f1421157c4db916e2523c715ef58d7a9e3784385576
Jul 23 00:18:48.569988 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.569958    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:18:48.692617 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.690611    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4k85j"]
Jul 23 00:18:48.694360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.694329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:48.699749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.699726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4k85j"]
Jul 23 00:18:48.822020 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.821960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttlc\" (UniqueName: \"kubernetes.io/projected/85243c52-df37-473e-9894-edaaf5aaf91d-kube-api-access-vttlc\") pod \"kuadrant-operator-catalog-4k85j\" (UID: \"85243c52-df37-473e-9894-edaaf5aaf91d\") " pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:48.922853 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.922818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttlc\" (UniqueName: \"kubernetes.io/projected/85243c52-df37-473e-9894-edaaf5aaf91d-kube-api-access-vttlc\") pod \"kuadrant-operator-catalog-4k85j\" (UID: \"85243c52-df37-473e-9894-edaaf5aaf91d\") " pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:48.937085 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:48.937049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttlc\" (UniqueName: \"kubernetes.io/projected/85243c52-df37-473e-9894-edaaf5aaf91d-kube-api-access-vttlc\") pod \"kuadrant-operator-catalog-4k85j\" (UID: \"85243c52-df37-473e-9894-edaaf5aaf91d\") " pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:49.004862 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:49.004831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:49.111948 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:49.111915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" event={"ID":"2a3fd8ee-346a-4805-bedf-d63d963aa40c","Type":"ContainerStarted","Data":"ffdf2d153d38110ba1b07f1421157c4db916e2523c715ef58d7a9e3784385576"}
Jul 23 00:18:49.133110 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:49.133082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4k85j"]
Jul 23 00:18:49.135020 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:18:49.134988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85243c52_df37_473e_9894_edaaf5aaf91d.slice/crio-d986710c7ae05072dc9a215a9f8a994442f1efff3fb3b00c697fd256c003f15e WatchSource:0}: Error finding container d986710c7ae05072dc9a215a9f8a994442f1efff3fb3b00c697fd256c003f15e: Status 404 returned error can't find the container with id d986710c7ae05072dc9a215a9f8a994442f1efff3fb3b00c697fd256c003f15e
Jul 23 00:18:50.119583 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:50.119547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4k85j" event={"ID":"85243c52-df37-473e-9894-edaaf5aaf91d","Type":"ContainerStarted","Data":"d986710c7ae05072dc9a215a9f8a994442f1efff3fb3b00c697fd256c003f15e"}
Jul 23 00:18:51.125911 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.125868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" event={"ID":"2a3fd8ee-346a-4805-bedf-d63d963aa40c","Type":"ContainerStarted","Data":"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"}
Jul 23 00:18:51.126385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.125951    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" podUID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" containerName="registry-server" containerID="cri-o://2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1" gracePeriod=2
Jul 23 00:18:51.127563 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.127501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4k85j" event={"ID":"85243c52-df37-473e-9894-edaaf5aaf91d","Type":"ContainerStarted","Data":"35993e7610f55758a015a64dc1b0aef84c8b331809bc06e9e949f28bd47de401"}
Jul 23 00:18:51.142126 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.142086    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" podStartSLOduration=0.979128192 podStartE2EDuration="3.142070569s" totalImagesPullingTime="2.162942377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:18:51.140268372 +0000 UTC m=+665.993838046" watchObservedRunningTime="2026-07-23 00:18:51.142070569 +0000 UTC m=+665.995640277"
Jul 23 00:18:51.155517 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.155472    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4k85j" podStartSLOduration=1.556138646 podStartE2EDuration="3.155458722s" totalImagesPullingTime="1.599320076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:18:51.153636136 +0000 UTC m=+666.007205809" watchObservedRunningTime="2026-07-23 00:18:51.155458722 +0000 UTC m=+666.009028401"
Jul 23 00:18:51.363642 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.363619    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:51.445214 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.445119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl66p\" (UniqueName: \"kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p\") pod \"2a3fd8ee-346a-4805-bedf-d63d963aa40c\" (UID: \"2a3fd8ee-346a-4805-bedf-d63d963aa40c\") "
Jul 23 00:18:51.447206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.447175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p" (OuterVolumeSpecName: "kube-api-access-sl66p") pod "2a3fd8ee-346a-4805-bedf-d63d963aa40c" (UID: "2a3fd8ee-346a-4805-bedf-d63d963aa40c"). InnerVolumeSpecName "kube-api-access-sl66p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:18:51.545930 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:51.545894    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl66p\" (UniqueName: \"kubernetes.io/projected/2a3fd8ee-346a-4805-bedf-d63d963aa40c-kube-api-access-sl66p\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:18:52.132625 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.132591    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" containerID="2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1" exitCode=0
Jul 23 00:18:52.133065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.132635    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mltc9"
Jul 23 00:18:52.133065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.132681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" event={"ID":"2a3fd8ee-346a-4805-bedf-d63d963aa40c","Type":"ContainerDied","Data":"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"}
Jul 23 00:18:52.133065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.132728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mltc9" event={"ID":"2a3fd8ee-346a-4805-bedf-d63d963aa40c","Type":"ContainerDied","Data":"ffdf2d153d38110ba1b07f1421157c4db916e2523c715ef58d7a9e3784385576"}
Jul 23 00:18:52.133065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.132758    2565 scope.go:117] "RemoveContainer" containerID="2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"
Jul 23 00:18:52.141273 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.141252    2565 scope.go:117] "RemoveContainer" containerID="2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"
Jul 23 00:18:52.141509 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:18:52.141494    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1\": container with ID starting with 2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1 not found: ID does not exist" containerID="2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"
Jul 23 00:18:52.141552 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.141516    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1"} err="failed to get container status \"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1\": rpc error: code = NotFound desc = could not find container \"2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1\": container with ID starting with 2d9e1589c187c9400557abbdfb4fd4cc57678008e8e7c1fd22678f19d24086f1 not found: ID does not exist"
Jul 23 00:18:52.148037 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.147956    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:52.153300 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:52.153280    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mltc9"]
Jul 23 00:18:53.724618 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:53.724585    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" path="/var/lib/kubelet/pods/2a3fd8ee-346a-4805-bedf-d63d963aa40c/volumes"
Jul 23 00:18:59.005210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:59.005167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:59.005612 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:59.005226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:59.027221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:59.027191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:18:59.180774 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:18:59.180748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4k85j"
Jul 23 00:19:19.916037 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.915997    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"]
Jul 23 00:19:19.916493 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.916403    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" containerName="registry-server"
Jul 23 00:19:19.916493 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.916423    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" containerName="registry-server"
Jul 23 00:19:19.916611 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.916509    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3fd8ee-346a-4805-bedf-d63d963aa40c" containerName="registry-server"
Jul 23 00:19:19.923677 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.923655    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:19.926684 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.926661    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m7gt7\""
Jul 23 00:19:19.926850 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.926837    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 00:19:19.935522 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:19.935497    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"]
Jul 23 00:19:20.075349 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:20.075309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mnz\" (UniqueName: \"kubernetes.io/projected/fa017948-914d-4a1b-ab3c-ef8b8e38244d-kube-api-access-d8mnz\") pod \"dns-operator-controller-manager-864dd78c4c-xq8nm\" (UID: \"fa017948-914d-4a1b-ab3c-ef8b8e38244d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:20.175933 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:20.175849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mnz\" (UniqueName: \"kubernetes.io/projected/fa017948-914d-4a1b-ab3c-ef8b8e38244d-kube-api-access-d8mnz\") pod \"dns-operator-controller-manager-864dd78c4c-xq8nm\" (UID: \"fa017948-914d-4a1b-ab3c-ef8b8e38244d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:20.184749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:20.184716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mnz\" (UniqueName: \"kubernetes.io/projected/fa017948-914d-4a1b-ab3c-ef8b8e38244d-kube-api-access-d8mnz\") pod \"dns-operator-controller-manager-864dd78c4c-xq8nm\" (UID: \"fa017948-914d-4a1b-ab3c-ef8b8e38244d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:20.233717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:20.233676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:20.360359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:20.360336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"]
Jul 23 00:19:20.362270 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:19:20.362245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa017948_914d_4a1b_ab3c_ef8b8e38244d.slice/crio-5171ce2cf58a16588a0733e26e47e1b268f4a372d2d9f1b062e738d9d9ac0921 WatchSource:0}: Error finding container 5171ce2cf58a16588a0733e26e47e1b268f4a372d2d9f1b062e738d9d9ac0921: Status 404 returned error can't find the container with id 5171ce2cf58a16588a0733e26e47e1b268f4a372d2d9f1b062e738d9d9ac0921
Jul 23 00:19:21.250927 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:21.250891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm" event={"ID":"fa017948-914d-4a1b-ab3c-ef8b8e38244d","Type":"ContainerStarted","Data":"5171ce2cf58a16588a0733e26e47e1b268f4a372d2d9f1b062e738d9d9ac0921"}
Jul 23 00:19:22.256507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:22.256469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm" event={"ID":"fa017948-914d-4a1b-ab3c-ef8b8e38244d","Type":"ContainerStarted","Data":"c6f1f164a1ebb43d5457dcdc8edb66e0bb6148a815bdc7f140bca61bd72381f1"}
Jul 23 00:19:22.256904 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:22.256580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:22.275182 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:22.275084    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm" podStartSLOduration=1.623179533 podStartE2EDuration="3.275066732s" totalImagesPullingTime="1.651887199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:22.273433139 +0000 UTC m=+697.127002813" watchObservedRunningTime="2026-07-23 00:19:22.275066732 +0000 UTC m=+697.128636476"
Jul 23 00:19:23.975308 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:23.975276    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"]
Jul 23 00:19:23.978471 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:23.978454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:23.980949 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:23.980926    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g6rzv\""
Jul 23 00:19:23.989879 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:23.989844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"]
Jul 23 00:19:24.106785 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:24.106727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw2l\" (UniqueName: \"kubernetes.io/projected/2981629d-879c-47ea-8b36-ed6429d2d2a1-kube-api-access-6sw2l\") pod \"authorino-operator-75cc8d5cb7-5xfjz\" (UID: \"2981629d-879c-47ea-8b36-ed6429d2d2a1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:24.208064 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:24.208021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw2l\" (UniqueName: \"kubernetes.io/projected/2981629d-879c-47ea-8b36-ed6429d2d2a1-kube-api-access-6sw2l\") pod \"authorino-operator-75cc8d5cb7-5xfjz\" (UID: \"2981629d-879c-47ea-8b36-ed6429d2d2a1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:24.218628 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:24.218601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw2l\" (UniqueName: \"kubernetes.io/projected/2981629d-879c-47ea-8b36-ed6429d2d2a1-kube-api-access-6sw2l\") pod \"authorino-operator-75cc8d5cb7-5xfjz\" (UID: \"2981629d-879c-47ea-8b36-ed6429d2d2a1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:24.294109 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:24.294079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:24.439036 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:24.439011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"]
Jul 23 00:19:24.441165 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:19:24.441133    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2981629d_879c_47ea_8b36_ed6429d2d2a1.slice/crio-ac3ca189163b568232ca1147bda9458351a4e3b1483a2b32616910d8151b525d WatchSource:0}: Error finding container ac3ca189163b568232ca1147bda9458351a4e3b1483a2b32616910d8151b525d: Status 404 returned error can't find the container with id ac3ca189163b568232ca1147bda9458351a4e3b1483a2b32616910d8151b525d
Jul 23 00:19:25.272025 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:25.271958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz" event={"ID":"2981629d-879c-47ea-8b36-ed6429d2d2a1","Type":"ContainerStarted","Data":"ac3ca189163b568232ca1147bda9458351a4e3b1483a2b32616910d8151b525d"}
Jul 23 00:19:27.282425 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.282392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz" event={"ID":"2981629d-879c-47ea-8b36-ed6429d2d2a1","Type":"ContainerStarted","Data":"a165d4387240f816784c268394daadc46f8334d9f1319fdddff6ed3ec213ebd8"}
Jul 23 00:19:27.282812 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.282532    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:27.314401 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.314341    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz" podStartSLOduration=1.984620858 podStartE2EDuration="4.31432346s" totalImagesPullingTime="2.329702602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:27.31264573 +0000 UTC m=+702.166215402" watchObservedRunningTime="2026-07-23 00:19:27.31432346 +0000 UTC m=+702.167893133"
Jul 23 00:19:27.366085 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.366051    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"]
Jul 23 00:19:27.369295 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.369275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:27.372227 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.372210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8zc48\""
Jul 23 00:19:27.383678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.383651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"]
Jul 23 00:19:27.534305 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.534219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmn6\" (UniqueName: \"kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" (UID: \"76e3c8bd-42ec-4755-a96a-679127824626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:27.635422 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.635375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmn6\" (UniqueName: \"kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" (UID: \"76e3c8bd-42ec-4755-a96a-679127824626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:27.643607 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.643584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmn6\" (UniqueName: \"kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6\") pod \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" (UID: \"76e3c8bd-42ec-4755-a96a-679127824626\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:27.679410 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.679381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:27.813261 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:27.813201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"]
Jul 23 00:19:27.815531 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:19:27.815507    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e3c8bd_42ec_4755_a96a_679127824626.slice/crio-c5ecca3eaf1b74f3d40b20781611ee99039cfdc890ec68c39018968c6bbe3ece WatchSource:0}: Error finding container c5ecca3eaf1b74f3d40b20781611ee99039cfdc890ec68c39018968c6bbe3ece: Status 404 returned error can't find the container with id c5ecca3eaf1b74f3d40b20781611ee99039cfdc890ec68c39018968c6bbe3ece
Jul 23 00:19:28.287892 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:28.287854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg" event={"ID":"76e3c8bd-42ec-4755-a96a-679127824626","Type":"ContainerStarted","Data":"c5ecca3eaf1b74f3d40b20781611ee99039cfdc890ec68c39018968c6bbe3ece"}
Jul 23 00:19:30.301223 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:30.301189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg" event={"ID":"76e3c8bd-42ec-4755-a96a-679127824626","Type":"ContainerStarted","Data":"bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03"}
Jul 23 00:19:30.301642 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:30.301298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:30.322744 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:30.322692    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg" podStartSLOduration=1.821579357 podStartE2EDuration="3.322677439s" totalImagesPullingTime="1.501098082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:30.320899702 +0000 UTC m=+705.174469376" watchObservedRunningTime="2026-07-23 00:19:30.322677439 +0000 UTC m=+705.176247180"
Jul 23 00:19:31.943063 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.943029    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"]
Jul 23 00:19:31.946500 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.946480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:31.949019 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.948997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rx7pw\""
Jul 23 00:19:31.957918 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.957897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"]
Jul 23 00:19:31.970663 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.970640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:31.970773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:31.970691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbb89\" (UniqueName: \"kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.071987 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.071940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.072147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.072043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb89\" (UniqueName: \"kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.072344 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.072317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.101163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.101130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb89\" (UniqueName: \"kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.256031 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.255965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:32.383114 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:32.383090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"]
Jul 23 00:19:32.385124 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:19:32.385097    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd47d19e_d021_4a7e_8818_21812c29d5c1.slice/crio-7d6f4af86630499c4033e8c76582c1d89320a0cf82347247cb6294c8096597d4 WatchSource:0}: Error finding container 7d6f4af86630499c4033e8c76582c1d89320a0cf82347247cb6294c8096597d4: Status 404 returned error can't find the container with id 7d6f4af86630499c4033e8c76582c1d89320a0cf82347247cb6294c8096597d4
Jul 23 00:19:33.263636 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:33.263603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xq8nm"
Jul 23 00:19:33.318040 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:33.318001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx" event={"ID":"fd47d19e-d021-4a7e-8818-21812c29d5c1","Type":"ContainerStarted","Data":"7d6f4af86630499c4033e8c76582c1d89320a0cf82347247cb6294c8096597d4"}
Jul 23 00:19:38.289915 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:38.289837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5xfjz"
Jul 23 00:19:38.346593 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:38.346554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx" event={"ID":"fd47d19e-d021-4a7e-8818-21812c29d5c1","Type":"ContainerStarted","Data":"265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0"}
Jul 23 00:19:38.346746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:38.346657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:38.367493 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:38.367449    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx" podStartSLOduration=2.272602079 podStartE2EDuration="7.367430976s" totalImagesPullingTime="5.094828897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:19:38.366190549 +0000 UTC m=+713.219760224" watchObservedRunningTime="2026-07-23 00:19:38.367430976 +0000 UTC m=+713.221000652"
Jul 23 00:19:40.112589 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.112554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"]
Jul 23 00:19:40.115204 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.115186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.117785 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.117763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 00:19:40.117915 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.117900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 00:19:40.118008 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.117921    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gfxkf\""
Jul 23 00:19:40.124730 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.124708    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"]
Jul 23 00:19:40.140716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.140690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.140825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.140762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hd8p\" (UniqueName: \"kubernetes.io/projected/7b310e1e-c3a8-48da-8804-c2c733991bfc-kube-api-access-7hd8p\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.140825 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.140816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b310e1e-c3a8-48da-8804-c2c733991bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.242183 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.242150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hd8p\" (UniqueName: \"kubernetes.io/projected/7b310e1e-c3a8-48da-8804-c2c733991bfc-kube-api-access-7hd8p\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.242346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.242210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b310e1e-c3a8-48da-8804-c2c733991bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.242346 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.242246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.242455 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:40.242360    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 23 00:19:40.242455 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:40.242419    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert podName:7b310e1e-c3a8-48da-8804-c2c733991bfc nodeName:}" failed. No retries permitted until 2026-07-23 00:19:40.742398794 +0000 UTC m=+715.595968461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rg5zn" (UID: "7b310e1e-c3a8-48da-8804-c2c733991bfc") : secret "plugin-serving-cert" not found
Jul 23 00:19:40.242868 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.242845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b310e1e-c3a8-48da-8804-c2c733991bfc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.250914 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.250892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hd8p\" (UniqueName: \"kubernetes.io/projected/7b310e1e-c3a8-48da-8804-c2c733991bfc-kube-api-access-7hd8p\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.745838 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.745804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:40.748135 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:40.748115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b310e1e-c3a8-48da-8804-c2c733991bfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rg5zn\" (UID: \"7b310e1e-c3a8-48da-8804-c2c733991bfc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:41.024879 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:41.024793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"
Jul 23 00:19:41.149421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:41.149397    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn"]
Jul 23 00:19:41.151291 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:19:41.151266    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b310e1e_c3a8_48da_8804_c2c733991bfc.slice/crio-e02e53b8884422a89f1e0b7db920db31d89bd8224f41c5faebace4da42cdd158 WatchSource:0}: Error finding container e02e53b8884422a89f1e0b7db920db31d89bd8224f41c5faebace4da42cdd158: Status 404 returned error can't find the container with id e02e53b8884422a89f1e0b7db920db31d89bd8224f41c5faebace4da42cdd158
Jul 23 00:19:41.307820 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:41.307750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:41.361239 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:41.361208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn" event={"ID":"7b310e1e-c3a8-48da-8804-c2c733991bfc","Type":"ContainerStarted","Data":"e02e53b8884422a89f1e0b7db920db31d89bd8224f41c5faebace4da42cdd158"}
Jul 23 00:19:49.352463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:49.352431    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:50.409108 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.409079    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"]
Jul 23 00:19:50.412378 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.412362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.426893 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.426866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"]
Jul 23 00:19:50.530157 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.530116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79659\" (UniqueName: \"kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.530332 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.530181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.631314 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.631272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79659\" (UniqueName: \"kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.631487 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.631335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.631693 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.631675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.639844 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.639821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79659\" (UniqueName: \"kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:50.723421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:50.723333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:19:51.145877 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.145840    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"]
Jul 23 00:19:51.149637 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.149608    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"]
Jul 23 00:19:51.164287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.164252    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"]
Jul 23 00:19:51.164496 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.164466    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" containerName="manager" containerID="cri-o://265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0" gracePeriod=2
Jul 23 00:19:51.168703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.168671    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:19:51.174927 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.174821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.175460 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.175384    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"]
Jul 23 00:19:51.189446 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.189420    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:19:51.189906 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.189886    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" containerName="manager"
Jul 23 00:19:51.189906 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.189907    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" containerName="manager"
Jul 23 00:19:51.190104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.190010    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" containerName="manager"
Jul 23 00:19:51.190861 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:51.190680    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fd47d19e-d021-4a7e-8818-21812c29d5c1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-546mx\": pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:51.193052 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:51.193014    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:51.193368 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.193338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"]
Jul 23 00:19:51.193457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.193390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:19:51.193522 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.193451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.193779 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.193680    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg" podUID="76e3c8bd-42ec-4755-a96a-679127824626" containerName="manager" containerID="cri-o://bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03" gracePeriod=2
Jul 23 00:19:51.202308 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.202224    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"]
Jul 23 00:19:51.205598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.205538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:19:51.225466 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:51.225426    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:19:51.235717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.235684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvzv\" (UniqueName: \"kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.235851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.235741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqrv\" (UniqueName: \"kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.235851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.235806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.235985 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.235855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.253876 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:19:51.253830    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:19:51.336668 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.336630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqrv\" (UniqueName: \"kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.336668 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.336673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.336905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.336704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.336905 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.336802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvzv\" (UniqueName: \"kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.337100 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.337071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.337187 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.337094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.345466 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.345438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqrv\" (UniqueName: \"kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fcskg\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:51.345609 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.345586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvzv\" (UniqueName: \"kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-qb7bq\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.576104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.576045    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:19:51.582950 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:51.582929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:19:52.306496 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:19:52.306449    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg" podUID="76e3c8bd-42ec-4755-a96a-679127824626" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 00:20:04.760870 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.760835    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:04.763576 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:04.763540    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:04.763717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.763612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:04.765750 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:04.765718    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:04.767781 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:04.767742    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:04.796277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.796249    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:20:04.797272 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:04.797244    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a10303e_cee5_43dd_9f3f_777c222b295e.slice/crio-23594a76faa4eb2e2c62e50e0d2959af1c41471e94f49c5e93d2db3320bf4da6 WatchSource:0}: Error finding container 23594a76faa4eb2e2c62e50e0d2959af1c41471e94f49c5e93d2db3320bf4da6: Status 404 returned error can't find the container with id 23594a76faa4eb2e2c62e50e0d2959af1c41471e94f49c5e93d2db3320bf4da6
Jul 23 00:20:04.811934 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.811890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:20:04.814034 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:04.814006    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003923d1_7795_4b21_b3ca_d5c9ca6c2d20.slice/crio-655119b651ce8fdd6214a2f132fcee8be023c81e51cc68f93c605d4849922d99 WatchSource:0}: Error finding container 655119b651ce8fdd6214a2f132fcee8be023c81e51cc68f93c605d4849922d99: Status 404 returned error can't find the container with id 655119b651ce8fdd6214a2f132fcee8be023c81e51cc68f93c605d4849922d99
Jul 23 00:20:04.862045 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.862020    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbb89\" (UniqueName: \"kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89\") pod \"fd47d19e-d021-4a7e-8818-21812c29d5c1\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") "
Jul 23 00:20:04.862141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.862061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume\") pod \"fd47d19e-d021-4a7e-8818-21812c29d5c1\" (UID: \"fd47d19e-d021-4a7e-8818-21812c29d5c1\") "
Jul 23 00:20:04.862276 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.862202    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rmn6\" (UniqueName: \"kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6\") pod \"76e3c8bd-42ec-4755-a96a-679127824626\" (UID: \"76e3c8bd-42ec-4755-a96a-679127824626\") "
Jul 23 00:20:04.862432 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.862410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fd47d19e-d021-4a7e-8818-21812c29d5c1" (UID: "fd47d19e-d021-4a7e-8818-21812c29d5c1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:20:04.863798 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.863779    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89" (OuterVolumeSpecName: "kube-api-access-cbb89") pod "fd47d19e-d021-4a7e-8818-21812c29d5c1" (UID: "fd47d19e-d021-4a7e-8818-21812c29d5c1"). InnerVolumeSpecName "kube-api-access-cbb89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:20:04.863944 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.863928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6" (OuterVolumeSpecName: "kube-api-access-4rmn6") pod "76e3c8bd-42ec-4755-a96a-679127824626" (UID: "76e3c8bd-42ec-4755-a96a-679127824626"). InnerVolumeSpecName "kube-api-access-4rmn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:20:04.963685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.963646    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbb89\" (UniqueName: \"kubernetes.io/projected/fd47d19e-d021-4a7e-8818-21812c29d5c1-kube-api-access-cbb89\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:04.963685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.963677    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd47d19e-d021-4a7e-8818-21812c29d5c1-extensions-socket-volume\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:04.963685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:04.963687    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rmn6\" (UniqueName: \"kubernetes.io/projected/76e3c8bd-42ec-4755-a96a-679127824626-kube-api-access-4rmn6\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:05.493070 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.493031    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd47d19e-d021-4a7e-8818-21812c29d5c1" containerID="265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0" exitCode=0
Jul 23 00:20:05.493309 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.493151    2565 scope.go:117] "RemoveContainer" containerID="265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0"
Jul 23 00:20:05.493403 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.493320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:05.495414 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.495386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn" event={"ID":"7b310e1e-c3a8-48da-8804-c2c733991bfc","Type":"ContainerStarted","Data":"ed0a84a938219db1b705eab0a22a5605cb10a129bb33f56bc7879221a15dbffc"}
Jul 23 00:20:05.495872 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.495814    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:05.497402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.497361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" event={"ID":"3a10303e-cee5-43dd-9f3f-777c222b295e","Type":"ContainerStarted","Data":"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"}
Jul 23 00:20:05.497495 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.497415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" event={"ID":"3a10303e-cee5-43dd-9f3f-777c222b295e","Type":"ContainerStarted","Data":"23594a76faa4eb2e2c62e50e0d2959af1c41471e94f49c5e93d2db3320bf4da6"}
Jul 23 00:20:05.497495 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.497436    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:20:05.497946 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.497921    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:05.499380 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.499347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" event={"ID":"003923d1-7795-4b21-b3ca-d5c9ca6c2d20","Type":"ContainerStarted","Data":"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"}
Jul 23 00:20:05.499550 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.499524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" event={"ID":"003923d1-7795-4b21-b3ca-d5c9ca6c2d20","Type":"ContainerStarted","Data":"655119b651ce8fdd6214a2f132fcee8be023c81e51cc68f93c605d4849922d99"}
Jul 23 00:20:05.499653 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.499564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:20:05.500143 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.500115    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:05.500791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.500767    2565 generic.go:356] "Generic (PLEG): container finished" podID="76e3c8bd-42ec-4755-a96a-679127824626" containerID="bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03" exitCode=0
Jul 23 00:20:05.500890 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.500813    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:05.502541 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.502508    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:05.503873 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.503853    2565 scope.go:117] "RemoveContainer" containerID="265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0"
Jul 23 00:20:05.504160 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.504139    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0\": container with ID starting with 265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0 not found: ID does not exist" containerID="265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0"
Jul 23 00:20:05.504241 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.504167    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0"} err="failed to get container status \"265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0\": rpc error: code = NotFound desc = could not find container \"265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0\": container with ID starting with 265dab0b3cbd89adf1742c677e15b2bc6195b9e0d28f895baff9217518c195a0 not found: ID does not exist"
Jul 23 00:20:05.504241 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.504184    2565 scope.go:117] "RemoveContainer" containerID="bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03"
Jul 23 00:20:05.513014 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.512995    2565 scope.go:117] "RemoveContainer" containerID="bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03"
Jul 23 00:20:05.513256 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.513239    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03\": container with ID starting with bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03 not found: ID does not exist" containerID="bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03"
Jul 23 00:20:05.513305 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.513264    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03"} err="failed to get container status \"bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03\": rpc error: code = NotFound desc = could not find container \"bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03\": container with ID starting with bdc567c616fc151e826f99244df9d83e4b66d2241b5e98f0854ce96df98e2b03 not found: ID does not exist"
Jul 23 00:20:05.518716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.518669    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rg5zn" podStartSLOduration=1.9952442590000001 podStartE2EDuration="25.518653707s" totalImagesPullingTime="23.523409448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:05.518309378 +0000 UTC m=+740.371879052" watchObservedRunningTime="2026-07-23 00:20:05.518653707 +0000 UTC m=+740.372223380"
Jul 23 00:20:05.536556 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.536509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" podStartSLOduration=14.536491919 podStartE2EDuration="14.536491919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:05.536156592 +0000 UTC m=+740.389726265" watchObservedRunningTime="2026-07-23 00:20:05.536491919 +0000 UTC m=+740.390061593"
Jul 23 00:20:05.538528 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.538492    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:05.558284 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.558241    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" podStartSLOduration=14.558225657 podStartE2EDuration="14.558225657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:05.557178235 +0000 UTC m=+740.410747909" watchObservedRunningTime="2026-07-23 00:20:05.558225657 +0000 UTC m=+740.411795329"
Jul 23 00:20:05.559474 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.559440    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:05.723874 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.723840    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-546mx\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-546mx"
Jul 23 00:20:05.725727 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.725693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e3c8bd-42ec-4755-a96a-679127824626" path="/var/lib/kubelet/pods/76e3c8bd-42ec-4755-a96a-679127824626/volumes"
Jul 23 00:20:05.726016 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:05.725993    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jwvjg\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="76e3c8bd-42ec-4755-a96a-679127824626" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jwvjg"
Jul 23 00:20:05.726092 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:05.726062    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd47d19e-d021-4a7e-8818-21812c29d5c1" path="/var/lib/kubelet/pods/fd47d19e-d021-4a7e-8818-21812c29d5c1/volumes"
Jul 23 00:20:07.162507 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:07.162462    2565 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-kszj5_kuadrant-system_e4c8d092-5657-4c91-96fe-c85bd948f443_0(1ab268f4953060673d0d59dcfc83fd3a2d5fbec30a652db030a2f2c095904b50): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-kszj5 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 23 00:20:07.162895 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:07.162527    2565 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-kszj5_kuadrant-system_e4c8d092-5657-4c91-96fe-c85bd948f443_0(1ab268f4953060673d0d59dcfc83fd3a2d5fbec30a652db030a2f2c095904b50): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-kszj5 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:07.511537 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.511500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:07.514480 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:07.514449    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="e4c8d092-5657-4c91-96fe-c85bd948f443" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:07.521394 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.521375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:07.524328 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:07.524308    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="e4c8d092-5657-4c91-96fe-c85bd948f443" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:07.689768 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.689732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79659\" (UniqueName: \"kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659\") pod \"e4c8d092-5657-4c91-96fe-c85bd948f443\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") "
Jul 23 00:20:07.689930 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.689791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume\") pod \"e4c8d092-5657-4c91-96fe-c85bd948f443\" (UID: \"e4c8d092-5657-4c91-96fe-c85bd948f443\") "
Jul 23 00:20:07.690105 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.690080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e4c8d092-5657-4c91-96fe-c85bd948f443" (UID: "e4c8d092-5657-4c91-96fe-c85bd948f443"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:20:07.691843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.691823    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659" (OuterVolumeSpecName: "kube-api-access-79659") pod "e4c8d092-5657-4c91-96fe-c85bd948f443" (UID: "e4c8d092-5657-4c91-96fe-c85bd948f443"). InnerVolumeSpecName "kube-api-access-79659". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:20:07.725130 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.725101    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c8d092-5657-4c91-96fe-c85bd948f443" path="/var/lib/kubelet/pods/e4c8d092-5657-4c91-96fe-c85bd948f443/volumes"
Jul 23 00:20:07.791255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.791177    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79659\" (UniqueName: \"kubernetes.io/projected/e4c8d092-5657-4c91-96fe-c85bd948f443-kube-api-access-79659\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:07.791255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:07.791208    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4c8d092-5657-4c91-96fe-c85bd948f443-extensions-socket-volume\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:08.515757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:08.515718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:08.518165 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:08.518133    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="e4c8d092-5657-4c91-96fe-c85bd948f443" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:08.520306 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:08.520281    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kszj5\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="e4c8d092-5657-4c91-96fe-c85bd948f443" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kszj5"
Jul 23 00:20:16.508805 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.508770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:20:16.509234 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.508846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:20:16.578247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.578217    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:20:16.578429 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.578391    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" podUID="3a10303e-cee5-43dd-9f3f-777c222b295e" containerName="manager" containerID="cri-o://f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840" gracePeriod=10
Jul 23 00:20:16.820322 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.820296    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:20:16.872647 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.872621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:20:16.872923 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.872912    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e3c8bd-42ec-4755-a96a-679127824626" containerName="manager"
Jul 23 00:20:16.872980 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.872925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e3c8bd-42ec-4755-a96a-679127824626" containerName="manager"
Jul 23 00:20:16.872980 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.872933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a10303e-cee5-43dd-9f3f-777c222b295e" containerName="manager"
Jul 23 00:20:16.872980 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.872940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a10303e-cee5-43dd-9f3f-777c222b295e" containerName="manager"
Jul 23 00:20:16.873074 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.873013    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e3c8bd-42ec-4755-a96a-679127824626" containerName="manager"
Jul 23 00:20:16.873074 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.873022    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a10303e-cee5-43dd-9f3f-777c222b295e" containerName="manager"
Jul 23 00:20:16.874835 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.874819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:16.897176 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.897151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:20:16.970498 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.970467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume\") pod \"3a10303e-cee5-43dd-9f3f-777c222b295e\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") "
Jul 23 00:20:16.970683 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.970540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spqrv\" (UniqueName: \"kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv\") pod \"3a10303e-cee5-43dd-9f3f-777c222b295e\" (UID: \"3a10303e-cee5-43dd-9f3f-777c222b295e\") "
Jul 23 00:20:16.970886 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.970862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a10303e-cee5-43dd-9f3f-777c222b295e" (UID: "3a10303e-cee5-43dd-9f3f-777c222b295e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:20:16.972561 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:16.972530    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv" (OuterVolumeSpecName: "kube-api-access-spqrv") pod "3a10303e-cee5-43dd-9f3f-777c222b295e" (UID: "3a10303e-cee5-43dd-9f3f-777c222b295e"). InnerVolumeSpecName "kube-api-access-spqrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:20:17.071553 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.071476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkvd\" (UniqueName: \"kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.071553 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.071541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.071727 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.071627    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqrv\" (UniqueName: \"kubernetes.io/projected/3a10303e-cee5-43dd-9f3f-777c222b295e-kube-api-access-spqrv\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:17.071727 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.071640    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a10303e-cee5-43dd-9f3f-777c222b295e-extensions-socket-volume\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:17.172437 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.172393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.172549 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.172461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkvd\" (UniqueName: \"kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.172778 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.172758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.183490 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.183469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkvd\" (UniqueName: \"kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd\") pod \"kuadrant-operator-controller-manager-86878c769f-x48k9\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.184247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.184229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.311532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.311509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:20:17.313640 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:17.313606    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16174b30_12f9_45cd_a1d4_88064961f598.slice/crio-d026a13d86f25f47b29805f40538674f2701fd32c32f796577f6a4c3820e3fb3 WatchSource:0}: Error finding container d026a13d86f25f47b29805f40538674f2701fd32c32f796577f6a4c3820e3fb3: Status 404 returned error can't find the container with id d026a13d86f25f47b29805f40538674f2701fd32c32f796577f6a4c3820e3fb3
Jul 23 00:20:17.556644 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.556605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" event={"ID":"16174b30-12f9-45cd-a1d4-88064961f598","Type":"ContainerStarted","Data":"e27e3b6c499357c946cfbaa5b6f6412ce33b7668b1661596d10e94e28b31d879"}
Jul 23 00:20:17.557087 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.556664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" event={"ID":"16174b30-12f9-45cd-a1d4-88064961f598","Type":"ContainerStarted","Data":"d026a13d86f25f47b29805f40538674f2701fd32c32f796577f6a4c3820e3fb3"}
Jul 23 00:20:17.557087 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.556706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:17.557923 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.557902    2565 generic.go:356] "Generic (PLEG): container finished" podID="3a10303e-cee5-43dd-9f3f-777c222b295e" containerID="f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840" exitCode=0
Jul 23 00:20:17.558054 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.557937    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"
Jul 23 00:20:17.558054 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.557952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" event={"ID":"3a10303e-cee5-43dd-9f3f-777c222b295e","Type":"ContainerDied","Data":"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"}
Jul 23 00:20:17.558054 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.558025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg" event={"ID":"3a10303e-cee5-43dd-9f3f-777c222b295e","Type":"ContainerDied","Data":"23594a76faa4eb2e2c62e50e0d2959af1c41471e94f49c5e93d2db3320bf4da6"}
Jul 23 00:20:17.558225 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.558054    2565 scope.go:117] "RemoveContainer" containerID="f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"
Jul 23 00:20:17.566391 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.566374    2565 scope.go:117] "RemoveContainer" containerID="f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"
Jul 23 00:20:17.566650 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:17.566631    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840\": container with ID starting with f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840 not found: ID does not exist" containerID="f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"
Jul 23 00:20:17.566695 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.566657    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840"} err="failed to get container status \"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840\": rpc error: code = NotFound desc = could not find container \"f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840\": container with ID starting with f6981c38fbc004653be60db8c43f3f1dda87e99eddad996c8a1dfe7d3342c840 not found: ID does not exist"
Jul 23 00:20:17.578151 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.578112    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" podStartSLOduration=1.578099431 podStartE2EDuration="1.578099431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:17.575991989 +0000 UTC m=+752.429561663" watchObservedRunningTime="2026-07-23 00:20:17.578099431 +0000 UTC m=+752.431669104"
Jul 23 00:20:17.589235 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.589175    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:20:17.593173 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.593150    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fcskg"]
Jul 23 00:20:17.725044 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:17.725008    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a10303e-cee5-43dd-9f3f-777c222b295e" path="/var/lib/kubelet/pods/3a10303e-cee5-43dd-9f3f-777c222b295e/volumes"
Jul 23 00:20:28.566341 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:28.566308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:20:28.608268 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:28.608239    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:20:28.608516 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:28.608468    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" podUID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" containerName="manager" containerID="cri-o://83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883" gracePeriod=10
Jul 23 00:20:29.411551 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.411523    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:20:29.459618 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.459588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume\") pod \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") "
Jul 23 00:20:29.459933 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.459912    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "003923d1-7795-4b21-b3ca-d5c9ca6c2d20" (UID: "003923d1-7795-4b21-b3ca-d5c9ca6c2d20"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:20:29.560558 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.560480    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcvzv\" (UniqueName: \"kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv\") pod \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\" (UID: \"003923d1-7795-4b21-b3ca-d5c9ca6c2d20\") "
Jul 23 00:20:29.560701 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.560668    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-extensions-socket-volume\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:29.562460 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.562423    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv" (OuterVolumeSpecName: "kube-api-access-rcvzv") pod "003923d1-7795-4b21-b3ca-d5c9ca6c2d20" (UID: "003923d1-7795-4b21-b3ca-d5c9ca6c2d20"). InnerVolumeSpecName "kube-api-access-rcvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:20:29.613860 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.613829    2565 generic.go:356] "Generic (PLEG): container finished" podID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" containerID="83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883" exitCode=0
Jul 23 00:20:29.614312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.613888    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"
Jul 23 00:20:29.614312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.613896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" event={"ID":"003923d1-7795-4b21-b3ca-d5c9ca6c2d20","Type":"ContainerDied","Data":"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"}
Jul 23 00:20:29.614312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.613953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq" event={"ID":"003923d1-7795-4b21-b3ca-d5c9ca6c2d20","Type":"ContainerDied","Data":"655119b651ce8fdd6214a2f132fcee8be023c81e51cc68f93c605d4849922d99"}
Jul 23 00:20:29.614312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.614004    2565 scope.go:117] "RemoveContainer" containerID="83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"
Jul 23 00:20:29.623264 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.623248    2565 scope.go:117] "RemoveContainer" containerID="83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"
Jul 23 00:20:29.623524 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:20:29.623506    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883\": container with ID starting with 83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883 not found: ID does not exist" containerID="83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"
Jul 23 00:20:29.623582 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.623534    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883"} err="failed to get container status \"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883\": rpc error: code = NotFound desc = could not find container \"83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883\": container with ID starting with 83a80a58d9d69f1c419e83e37ba1a96828d0e8a8db61c6786bcf0338f6d4a883 not found: ID does not exist"
Jul 23 00:20:29.636272 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.636248    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:20:29.641051 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.641031    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-qb7bq"]
Jul 23 00:20:29.661853 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.661828    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcvzv\" (UniqueName: \"kubernetes.io/projected/003923d1-7795-4b21-b3ca-d5c9ca6c2d20-kube-api-access-rcvzv\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:20:29.725671 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:29.725641    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" path="/var/lib/kubelet/pods/003923d1-7795-4b21-b3ca-d5c9ca6c2d20/volumes"
Jul 23 00:20:33.014279 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.014238    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"]
Jul 23 00:20:33.014634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.014580    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" containerName="manager"
Jul 23 00:20:33.014634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.014593    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" containerName="manager"
Jul 23 00:20:33.014707 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.014639    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="003923d1-7795-4b21-b3ca-d5c9ca6c2d20" containerName="manager"
Jul 23 00:20:33.128594 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.128557    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"]
Jul 23 00:20:33.128734 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.128679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.131268 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.131249    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sd59p\""
Jul 23 00:20:33.131391 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.131249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 00:20:33.184823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.184823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.184963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.185021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f310acc-20bd-4048-bf4f-a08ace8d187d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.185064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.185206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.185084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdd56\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-kube-api-access-zdd56\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f310acc-20bd-4048-bf4f-a08ace8d187d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdd56\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-kube-api-access-zdd56\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286850 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.286850 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.287004 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.286984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.287078 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.287058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f310acc-20bd-4048-bf4f-a08ace8d187d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.287204 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.287183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.288764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.288740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.288856 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.288840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.294526 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.294502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.294648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.294631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdd56\" (UniqueName: \"kubernetes.io/projected/8f310acc-20bd-4048-bf4f-a08ace8d187d-kube-api-access-zdd56\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6fnhl\" (UID: \"8f310acc-20bd-4048-bf4f-a08ace8d187d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.439243 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.439203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:33.661229 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:33.661200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"]
Jul 23 00:20:33.664275 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:33.664245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f310acc_20bd_4048_bf4f_a08ace8d187d.slice/crio-fe01eddb04096d54419805395971bad82fb3a0beea585e0cb3b609082d906f6c WatchSource:0}: Error finding container fe01eddb04096d54419805395971bad82fb3a0beea585e0cb3b609082d906f6c: Status 404 returned error can't find the container with id fe01eddb04096d54419805395971bad82fb3a0beea585e0cb3b609082d906f6c
Jul 23 00:20:34.639337 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:34.639302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" event={"ID":"8f310acc-20bd-4048-bf4f-a08ace8d187d","Type":"ContainerStarted","Data":"fe01eddb04096d54419805395971bad82fb3a0beea585e0cb3b609082d906f6c"}
Jul 23 00:20:36.421389 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:36.421352    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:20:36.421769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:36.421435    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:20:36.421769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:36.421484    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:20:36.650552 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:36.650514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" event={"ID":"8f310acc-20bd-4048-bf4f-a08ace8d187d","Type":"ContainerStarted","Data":"b57ed8d80fd629ca64bccf52f21d374c69acd07aff8d9eb8bf1ea839b059c63d"}
Jul 23 00:20:36.675079 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:36.674958    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podStartSLOduration=1.920070667 podStartE2EDuration="4.674935555s" totalImagesPullingTime="2.754864888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:36.671813871 +0000 UTC m=+771.525383545" watchObservedRunningTime="2026-07-23 00:20:36.674935555 +0000 UTC m=+771.528505229"
Jul 23 00:20:37.440035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:37.439989    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:37.444598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:37.444576    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:37.655411 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:37.655378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:37.656426 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:37.656408    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:20:44.070440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.070399    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:20:44.072779 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.072763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.075415 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.075394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 00:20:44.082012 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.081985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:20:44.164534 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.164504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:20:44.176370 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.176341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.176510 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.176408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.276844 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.276815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.277035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.276864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.277446 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.277428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.285714 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.285683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6\") pod \"limitador-limitador-64c94fd64f-jgnxn\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.383364 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.383291    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:44.463580 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.463552    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:20:44.466427 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.466177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.478786 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.478762    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:20:44.491721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.491688    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:20:44.515343 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.515323    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:20:44.517420 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:44.517392    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df18275_fca6_4a9d_88f2_cb5cca10e51e.slice/crio-e1290f9b0eb4cd6e43d4a3d5c3695d11186773b2f69a841e87a411c4b8046d84 WatchSource:0}: Error finding container e1290f9b0eb4cd6e43d4a3d5c3695d11186773b2f69a841e87a411c4b8046d84: Status 404 returned error can't find the container with id e1290f9b0eb4cd6e43d4a3d5c3695d11186773b2f69a841e87a411c4b8046d84
Jul 23 00:20:44.578569 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.578541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48113520-45cc-4a33-840b-733c0a348870-config-file\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.578707 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.578585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgtj\" (UniqueName: \"kubernetes.io/projected/48113520-45cc-4a33-840b-733c0a348870-kube-api-access-bqgtj\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.679603 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.679506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48113520-45cc-4a33-840b-733c0a348870-config-file\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.679603 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.679559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgtj\" (UniqueName: \"kubernetes.io/projected/48113520-45cc-4a33-840b-733c0a348870-kube-api-access-bqgtj\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.680241 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.680222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/48113520-45cc-4a33-840b-733c0a348870-config-file\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.688336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.688306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" event={"ID":"4df18275-fca6-4a9d-88f2-cb5cca10e51e","Type":"ContainerStarted","Data":"e1290f9b0eb4cd6e43d4a3d5c3695d11186773b2f69a841e87a411c4b8046d84"}
Jul 23 00:20:44.688730 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.688709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgtj\" (UniqueName: \"kubernetes.io/projected/48113520-45cc-4a33-840b-733c0a348870-kube-api-access-bqgtj\") pod \"limitador-limitador-54d6d988d5-bcvbx\" (UID: \"48113520-45cc-4a33-840b-733c0a348870\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.778511 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.778473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:44.903472 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:44.903449    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:20:44.905873 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:20:44.905844    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48113520_45cc_4a33_840b_733c0a348870.slice/crio-870ddf6c910cd9fb5ba7c519939d15967a356cea6b9414688d31683e106d9dea WatchSource:0}: Error finding container 870ddf6c910cd9fb5ba7c519939d15967a356cea6b9414688d31683e106d9dea: Status 404 returned error can't find the container with id 870ddf6c910cd9fb5ba7c519939d15967a356cea6b9414688d31683e106d9dea
Jul 23 00:20:45.699293 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:45.699254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx" event={"ID":"48113520-45cc-4a33-840b-733c0a348870","Type":"ContainerStarted","Data":"870ddf6c910cd9fb5ba7c519939d15967a356cea6b9414688d31683e106d9dea"}
Jul 23 00:20:48.715795 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.715750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx" event={"ID":"48113520-45cc-4a33-840b-733c0a348870","Type":"ContainerStarted","Data":"d51b8d3c322d5fa9d76a2ed79d1dbd839484113a994ba7d656ee23967b899908"}
Jul 23 00:20:48.716231 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.715839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:48.717480 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.717453    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" event={"ID":"4df18275-fca6-4a9d-88f2-cb5cca10e51e","Type":"ContainerStarted","Data":"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"}
Jul 23 00:20:48.717620 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.717553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:48.732407 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.732367    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx" podStartSLOduration=1.8704083950000001 podStartE2EDuration="4.73235465s" totalImagesPullingTime="2.861946255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:48.73053584 +0000 UTC m=+783.584105513" watchObservedRunningTime="2026-07-23 00:20:48.73235465 +0000 UTC m=+783.585924322"
Jul 23 00:20:48.745127 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:48.745090    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" podStartSLOduration=1.484778962 podStartE2EDuration="4.745078498s" totalImagesPullingTime="3.260299536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:20:48.744611759 +0000 UTC m=+783.598181433" watchObservedRunningTime="2026-07-23 00:20:48.745078498 +0000 UTC m=+783.598648194"
Jul 23 00:20:59.725197 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:59.725169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:20:59.725591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:59.725228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"
Jul 23 00:20:59.783020 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:59.782988    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:20:59.783206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:20:59.783166    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" podUID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" containerName="limitador" containerID="cri-o://75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca" gracePeriod=30
Jul 23 00:21:00.730551 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.730527    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:21:00.772555 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.772516    2565 generic.go:356] "Generic (PLEG): container finished" podID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" containerID="75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca" exitCode=0
Jul 23 00:21:00.772689 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.772581    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"
Jul 23 00:21:00.772689 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.772594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" event={"ID":"4df18275-fca6-4a9d-88f2-cb5cca10e51e","Type":"ContainerDied","Data":"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"}
Jul 23 00:21:00.772689 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.772653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgnxn" event={"ID":"4df18275-fca6-4a9d-88f2-cb5cca10e51e","Type":"ContainerDied","Data":"e1290f9b0eb4cd6e43d4a3d5c3695d11186773b2f69a841e87a411c4b8046d84"}
Jul 23 00:21:00.772689 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.772677    2565 scope.go:117] "RemoveContainer" containerID="75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"
Jul 23 00:21:00.781309 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.781288    2565 scope.go:117] "RemoveContainer" containerID="75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"
Jul 23 00:21:00.781556 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:21:00.781537    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca\": container with ID starting with 75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca not found: ID does not exist" containerID="75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"
Jul 23 00:21:00.781622 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.781565    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca"} err="failed to get container status \"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca\": rpc error: code = NotFound desc = could not find container \"75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca\": container with ID starting with 75330ba4c37632e519fefc3d24839c39d4038161b36df85b385c050db60e66ca not found: ID does not exist"
Jul 23 00:21:00.818494 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.818468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file\") pod \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") "
Jul 23 00:21:00.818592 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.818517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6\") pod \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\" (UID: \"4df18275-fca6-4a9d-88f2-cb5cca10e51e\") "
Jul 23 00:21:00.818816 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.818796    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file" (OuterVolumeSpecName: "config-file") pod "4df18275-fca6-4a9d-88f2-cb5cca10e51e" (UID: "4df18275-fca6-4a9d-88f2-cb5cca10e51e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:21:00.820553 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.820535    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6" (OuterVolumeSpecName: "kube-api-access-4wtn6") pod "4df18275-fca6-4a9d-88f2-cb5cca10e51e" (UID: "4df18275-fca6-4a9d-88f2-cb5cca10e51e"). InnerVolumeSpecName "kube-api-access-4wtn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:21:00.919786 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.919701    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4df18275-fca6-4a9d-88f2-cb5cca10e51e-config-file\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:21:00.919786 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:00.919731    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtn6\" (UniqueName: \"kubernetes.io/projected/4df18275-fca6-4a9d-88f2-cb5cca10e51e-kube-api-access-4wtn6\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:21:01.093243 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:01.093210    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:21:01.099203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:01.099177    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgnxn"]
Jul 23 00:21:01.725051 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:01.725020    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" path="/var/lib/kubelet/pods/4df18275-fca6-4a9d-88f2-cb5cca10e51e/volumes"
Jul 23 00:21:23.760137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.760102    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:21:23.760565 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.760419    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" containerName="limitador"
Jul 23 00:21:23.760565 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.760429    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" containerName="limitador"
Jul 23 00:21:23.760565 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.760476    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df18275-fca6-4a9d-88f2-cb5cca10e51e" containerName="limitador"
Jul 23 00:21:23.765095 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.765075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:23.768859 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.768833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 00:21:23.768989 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.768910    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4t6cw\""
Jul 23 00:21:23.768989 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.768841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 00:21:23.772182 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.771895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:21:23.905272 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.905245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkfm\" (UniqueName: \"kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:23.905457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:23.905282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.006253 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.006226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkfm\" (UniqueName: \"kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.006403 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.006263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.008390 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.008367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.015392 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.015329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkfm\" (UniqueName: \"kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm\") pod \"maas-controller-5fccc86db5-dh25t\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") " pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.075894 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.075871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:24.200578 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.200552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:21:24.202119 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:21:24.202091    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcc2461_7753_46e7_b3c4_8f55f5bbb6f4.slice/crio-84cea36f3e48985e6f3c4709153cab318618574cbacfd8b7aa810d260ad12ee1 WatchSource:0}: Error finding container 84cea36f3e48985e6f3c4709153cab318618574cbacfd8b7aa810d260ad12ee1: Status 404 returned error can't find the container with id 84cea36f3e48985e6f3c4709153cab318618574cbacfd8b7aa810d260ad12ee1
Jul 23 00:21:24.879330 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:24.879243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-dh25t" event={"ID":"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4","Type":"ContainerStarted","Data":"84cea36f3e48985e6f3c4709153cab318618574cbacfd8b7aa810d260ad12ee1"}
Jul 23 00:21:26.890311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:26.890275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-dh25t" event={"ID":"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4","Type":"ContainerStarted","Data":"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"}
Jul 23 00:21:26.890710 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:26.890413    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:21:26.912898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:26.912854    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fccc86db5-dh25t" podStartSLOduration=1.738280637 podStartE2EDuration="3.91284174s" totalImagesPullingTime="2.174561103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:21:26.912171962 +0000 UTC m=+821.765741636" watchObservedRunningTime="2026-07-23 00:21:26.91284174 +0000 UTC m=+821.766411412"
Jul 23 00:21:37.898999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:21:37.898950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:22:01.103615 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.103577    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"]
Jul 23 00:22:01.112391 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.112363    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.115270 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.115244    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 00:22:01.115745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.115721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"]
Jul 23 00:22:01.128950 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.128929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:22:01.211158 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.211126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba58cb79-afcb-471f-88b2-edd213181d52-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.211158 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.211160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwth\" (UniqueName: \"kubernetes.io/projected/ba58cb79-afcb-471f-88b2-edd213181d52-kube-api-access-lcwth\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.312414 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.312377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba58cb79-afcb-471f-88b2-edd213181d52-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.312414 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.312424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwth\" (UniqueName: \"kubernetes.io/projected/ba58cb79-afcb-471f-88b2-edd213181d52-kube-api-access-lcwth\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.313115 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.313094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba58cb79-afcb-471f-88b2-edd213181d52-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.320315 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.320285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwth\" (UniqueName: \"kubernetes.io/projected/ba58cb79-afcb-471f-88b2-edd213181d52-kube-api-access-lcwth\") pod \"payload-pre-processing-7677b774bb-jdqt4\" (UID: \"ba58cb79-afcb-471f-88b2-edd213181d52\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.425147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.425064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:01.763585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:01.763562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"]
Jul 23 00:22:01.765888 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:22:01.765864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba58cb79_afcb_471f_88b2_edd213181d52.slice/crio-467946c70028942b860510de6541fc69248af296f267692df94a18f310175966 WatchSource:0}: Error finding container 467946c70028942b860510de6541fc69248af296f267692df94a18f310175966: Status 404 returned error can't find the container with id 467946c70028942b860510de6541fc69248af296f267692df94a18f310175966
Jul 23 00:22:02.045105 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:02.045020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4" event={"ID":"ba58cb79-afcb-471f-88b2-edd213181d52","Type":"ContainerStarted","Data":"467946c70028942b860510de6541fc69248af296f267692df94a18f310175966"}
Jul 23 00:22:04.055699 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:04.055662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4" event={"ID":"ba58cb79-afcb-471f-88b2-edd213181d52","Type":"ContainerStarted","Data":"46499505d7a64fcc06f090dfffe7eb52a939e65b6696109f7729809423437732"}
Jul 23 00:22:04.056115 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:04.055746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:04.073900 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:04.073834    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4" podStartSLOduration=0.950475946 podStartE2EDuration="3.073813388s" totalImagesPullingTime="2.123337442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:04.071567822 +0000 UTC m=+858.925137495" watchObservedRunningTime="2026-07-23 00:22:04.073813388 +0000 UTC m=+858.927383061"
Jul 23 00:22:08.519991 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:08.519940    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6fnhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:56804->10.132.0.33:15021: read: connection reset by peer" start-of-body=
Jul 23 00:22:08.520415 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:08.520041    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podUID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:56804->10.132.0.33:15021: read: connection reset by peer"
Jul 23 00:22:09.080347 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.080312    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerID="b57ed8d80fd629ca64bccf52f21d374c69acd07aff8d9eb8bf1ea839b059c63d" exitCode=0
Jul 23 00:22:09.080525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.080388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" event={"ID":"8f310acc-20bd-4048-bf4f-a08ace8d187d","Type":"ContainerDied","Data":"b57ed8d80fd629ca64bccf52f21d374c69acd07aff8d9eb8bf1ea839b059c63d"}
Jul 23 00:22:09.080781 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.080765    2565 scope.go:117] "RemoveContainer" containerID="b57ed8d80fd629ca64bccf52f21d374c69acd07aff8d9eb8bf1ea839b059c63d"
Jul 23 00:22:09.081761 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.081725    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:22:09.081855 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.081819    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:22:09.081919 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:09.081866    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:22:10.089587 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:10.089551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" event={"ID":"8f310acc-20bd-4048-bf4f-a08ace8d187d","Type":"ContainerStarted","Data":"09d50f679d06c947091cd7404cee68a5594f2eb5dec22482b6b5e679775b4a0d"}
Jul 23 00:22:10.439897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:10.439803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:22:10.440080 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:10.440058    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6fnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 23 00:22:10.440128 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:10.440105    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podUID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 23 00:22:11.440163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:11.440125    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6fnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 23 00:22:11.440559 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:11.440182    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podUID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 23 00:22:12.440284 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:12.440240    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6fnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 23 00:22:12.440693 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:12.440302    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podUID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 23 00:22:13.440223 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:13.440180    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6fnhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 23 00:22:13.440413 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:13.440239    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl" podUID="8f310acc-20bd-4048-bf4f-a08ace8d187d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 23 00:22:14.443274 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:14.443243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:22:14.443682 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:14.443617    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:22:14.444325 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:14.444306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6fnhl"
Jul 23 00:22:15.062997 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:15.062958    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-jdqt4"
Jul 23 00:22:26.610611 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.610564    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"]
Jul 23 00:22:26.614319 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.614294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.617075 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.617052    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jghlj\""
Jul 23 00:22:26.617190 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.617074    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 00:22:26.617190 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.617134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 00:22:26.618255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.618234    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 23 00:22:26.624812 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.624791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"]
Jul 23 00:22:26.737026 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.736996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.737205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.737052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.737205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.737147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.737205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.737181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtwj\" (UniqueName: \"kubernetes.io/projected/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-kube-api-access-hhtwj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.737369 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.737210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.737369 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.737284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838469 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838659 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838659 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtwj\" (UniqueName: \"kubernetes.io/projected/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-kube-api-access-hhtwj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838659 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838833 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838833 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838934 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.838934 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.838899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.839052 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.839031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.840892 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.840872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.841312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.841293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.847382 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.847360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtwj\" (UniqueName: \"kubernetes.io/projected/131a67ff-3ea8-4ab0-8cb4-265710c8c3f9-kube-api-access-hhtwj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr\" (UID: \"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:26.926291 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:26.926225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:27.063201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:27.063173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"]
Jul 23 00:22:27.065286 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:22:27.065258    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131a67ff_3ea8_4ab0_8cb4_265710c8c3f9.slice/crio-833c93e523579128bce881ddde090e9845096d4c12532659b9d9b95207681625 WatchSource:0}: Error finding container 833c93e523579128bce881ddde090e9845096d4c12532659b9d9b95207681625: Status 404 returned error can't find the container with id 833c93e523579128bce881ddde090e9845096d4c12532659b9d9b95207681625
Jul 23 00:22:27.116914 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:27.116879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:22:27.168136 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:27.168101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr" event={"ID":"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9","Type":"ContainerStarted","Data":"833c93e523579128bce881ddde090e9845096d4c12532659b9d9b95207681625"}
Jul 23 00:22:29.182241 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:29.182202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr" event={"ID":"131a67ff-3ea8-4ab0-8cb4-265710c8c3f9","Type":"ContainerStarted","Data":"e8bfd9e1a437c620f193ec82f88f3debabc5ffe50355fab03d5343f02935e0a1"}
Jul 23 00:22:29.200247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:29.200202    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr" podStartSLOduration=2.062431909 podStartE2EDuration="3.200187457s" totalImagesPullingTime="1.137755548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:29.198579299 +0000 UTC m=+884.052148972" watchObservedRunningTime="2026-07-23 00:22:29.200187457 +0000 UTC m=+884.053757130"
Jul 23 00:22:32.036601 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.036565    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"]
Jul 23 00:22:32.040585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.040564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.043357 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.043335    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 00:22:32.046641 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.046611    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"]
Jul 23 00:22:32.086638 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.086775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9x4w\" (UniqueName: \"kubernetes.io/projected/35c01219-0888-4e0e-a9a7-56fe45ded8a8-kube-api-access-p9x4w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.086775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.086775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.086775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.087004 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.086794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187306 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9x4w\" (UniqueName: \"kubernetes.io/projected/35c01219-0888-4e0e-a9a7-56fe45ded8a8-kube-api-access-p9x4w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187463 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187790 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.187869 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.187853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.189687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.189649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35c01219-0888-4e0e-a9a7-56fe45ded8a8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.189961 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.189939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35c01219-0888-4e0e-a9a7-56fe45ded8a8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.195555 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.195535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9x4w\" (UniqueName: \"kubernetes.io/projected/35c01219-0888-4e0e-a9a7-56fe45ded8a8-kube-api-access-p9x4w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xbktw\" (UID: \"35c01219-0888-4e0e-a9a7-56fe45ded8a8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.370665 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.370576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:32.503258 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.503213    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"]
Jul 23 00:22:32.505382 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:22:32.505358    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c01219_0888_4e0e_a9a7_56fe45ded8a8.slice/crio-3ebc806cbae4cd720e84a7019c92f652dd8505a0e85dbfe482284a8b38775c76 WatchSource:0}: Error finding container 3ebc806cbae4cd720e84a7019c92f652dd8505a0e85dbfe482284a8b38775c76: Status 404 returned error can't find the container with id 3ebc806cbae4cd720e84a7019c92f652dd8505a0e85dbfe482284a8b38775c76
Jul 23 00:22:32.822282 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:32.822243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:22:33.207702 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:33.207664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw" event={"ID":"35c01219-0888-4e0e-a9a7-56fe45ded8a8","Type":"ContainerStarted","Data":"d872c1749926701485f1fa25c3787411a1350145e4a183dbc0f766fdd339abf1"}
Jul 23 00:22:33.208113 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:33.207767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw" event={"ID":"35c01219-0888-4e0e-a9a7-56fe45ded8a8","Type":"ContainerStarted","Data":"3ebc806cbae4cd720e84a7019c92f652dd8505a0e85dbfe482284a8b38775c76"}
Jul 23 00:22:33.225737 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:33.225690    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw" podStartSLOduration=1.031979439 podStartE2EDuration="1.225676839s" totalImagesPullingTime="193.6974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:22:33.225066889 +0000 UTC m=+888.078636562" watchObservedRunningTime="2026-07-23 00:22:33.225676839 +0000 UTC m=+888.079246512"
Jul 23 00:22:36.926637 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:36.926596    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:36.927122 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:36.926690    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:36.939307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:36.939282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:37.236929 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:37.236848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr"
Jul 23 00:22:42.371581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:42.371497    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:42.371581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:42.371578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:42.384009 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:42.383986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:43.262838 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:43.262803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xbktw"
Jul 23 00:22:46.328756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:22:46.328719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:23:07.934667 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:07.934634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:23:16.224713 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:16.224678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:23:45.012493 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.012461    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"]
Jul 23 00:23:45.015800 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.015779    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.018528 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.018510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 00:23:45.023196 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.023170    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"]
Jul 23 00:23:45.069673 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.069646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29faca5d-19c5-43ee-bad1-f53d6132290a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.069908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.069682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqnp\" (UniqueName: \"kubernetes.io/projected/29faca5d-19c5-43ee-bad1-f53d6132290a-kube-api-access-lcqnp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.069908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.069706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.069908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.069855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.069908 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.069901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.070161 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.070054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.170924 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.170885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171132 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.170941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171132 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.170995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171132 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29faca5d-19c5-43ee-bad1-f53d6132290a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171132 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqnp\" (UniqueName: \"kubernetes.io/projected/29faca5d-19c5-43ee-bad1-f53d6132290a-kube-api-access-lcqnp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171132 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171423 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171570 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.171667 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.171644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.173386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.173359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29faca5d-19c5-43ee-bad1-f53d6132290a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.173933 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.173914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29faca5d-19c5-43ee-bad1-f53d6132290a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.179585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.179562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqnp\" (UniqueName: \"kubernetes.io/projected/29faca5d-19c5-43ee-bad1-f53d6132290a-kube-api-access-lcqnp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg\" (UID: \"29faca5d-19c5-43ee-bad1-f53d6132290a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.325578 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.325499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:45.455226 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.455200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"]
Jul 23 00:23:45.457176 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:23:45.457150    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29faca5d_19c5_43ee_bad1_f53d6132290a.slice/crio-9570095f8f14e76312e479ae6e294111322fce154dfadfccba2dd107de464825 WatchSource:0}: Error finding container 9570095f8f14e76312e479ae6e294111322fce154dfadfccba2dd107de464825: Status 404 returned error can't find the container with id 9570095f8f14e76312e479ae6e294111322fce154dfadfccba2dd107de464825
Jul 23 00:23:45.542530 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:45.542498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg" event={"ID":"29faca5d-19c5-43ee-bad1-f53d6132290a","Type":"ContainerStarted","Data":"9570095f8f14e76312e479ae6e294111322fce154dfadfccba2dd107de464825"}
Jul 23 00:23:46.547983 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:46.547934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg" event={"ID":"29faca5d-19c5-43ee-bad1-f53d6132290a","Type":"ContainerStarted","Data":"3fa46615ad8f716a329ad5345a21376a552a8f14eb22d0d4e1916143fe364d19"}
Jul 23 00:23:48.433820 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:48.433761    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg" podStartSLOduration=4.235813323 podStartE2EDuration="4.433745922s" totalImagesPullingTime="197.932599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:46.566213157 +0000 UTC m=+961.419782831" watchObservedRunningTime="2026-07-23 00:23:48.433745922 +0000 UTC m=+963.287315573"
Jul 23 00:23:48.434890 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:48.434865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:23:55.326281 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:55.326239    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:55.326746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:55.326301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:55.338665 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:55.338637    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:23:55.603159 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:23:55.603080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg"
Jul 23 00:25:14.035000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:14.034062    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:25:24.429895 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:24.429859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:25:32.689084 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:32.689046    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:25:32.689520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:32.689333    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fccc86db5-dh25t" podUID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" containerName="manager" containerID="cri-o://ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16" gracePeriod=30
Jul 23 00:25:32.937884 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:32.937859    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:25:32.999445 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:32.999413    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbkfm\" (UniqueName: \"kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm\") pod \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") "
Jul 23 00:25:32.999622 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:32.999451    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert\") pod \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\" (UID: \"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4\") "
Jul 23 00:25:33.001579 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.001552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" (UID: "6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:25:33.001694 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.001620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm" (OuterVolumeSpecName: "kube-api-access-bbkfm") pod "6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" (UID: "6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4"). InnerVolumeSpecName "kube-api-access-bbkfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:25:33.047907 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.047877    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" containerID="ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16" exitCode=0
Jul 23 00:25:33.048057 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.047918    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-dh25t"
Jul 23 00:25:33.048057 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.047962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-dh25t" event={"ID":"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4","Type":"ContainerDied","Data":"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"}
Jul 23 00:25:33.048057 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.048025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-dh25t" event={"ID":"6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4","Type":"ContainerDied","Data":"84cea36f3e48985e6f3c4709153cab318618574cbacfd8b7aa810d260ad12ee1"}
Jul 23 00:25:33.048057 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.048047    2565 scope.go:117] "RemoveContainer" containerID="ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"
Jul 23 00:25:33.062764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.062742    2565 scope.go:117] "RemoveContainer" containerID="ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"
Jul 23 00:25:33.063054 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:25:33.063035    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16\": container with ID starting with ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16 not found: ID does not exist" containerID="ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"
Jul 23 00:25:33.063108 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.063062    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16"} err="failed to get container status \"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16\": rpc error: code = NotFound desc = could not find container \"ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16\": container with ID starting with ccb9524ef044ee3b0b15cfb594a777789dc2cda9f43821406455fddcd8746a16 not found: ID does not exist"
Jul 23 00:25:33.072733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.072708    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:25:33.076375 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.076353    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-dh25t"]
Jul 23 00:25:33.100253 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.100231    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkfm\" (UniqueName: \"kubernetes.io/projected/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-kube-api-access-bbkfm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:25:33.100326 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.100254    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4-webhook-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:25:33.726177 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:33.726144    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" path="/var/lib/kubelet/pods/6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4/volumes"
Jul 23 00:25:34.428713 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:25:34.428676    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:26:33.910181 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.910145    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:33.910625 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.910492    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" containerName="manager"
Jul 23 00:26:33.910625 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.910503    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" containerName="manager"
Jul 23 00:26:33.910625 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.910563    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcc2461-7753-46e7-b3c4-8f55f5bbb6f4" containerName="manager"
Jul 23 00:26:33.913564 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.913544    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:33.916220 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.916195    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 00:26:33.916691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.916670    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 00:26:33.916794 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.916674    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4t6cw\""
Jul 23 00:26:33.922072 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:33.922049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:34.025793 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.025762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2p8\" (UniqueName: \"kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.025950 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.025801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.127252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.127217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2p8\" (UniqueName: \"kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.127412 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.127264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.129572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.129543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.135595 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.135571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2p8\" (UniqueName: \"kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8\") pod \"maas-controller-5fccc86db5-ft8j8\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") " pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.225564 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.225486    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:34.357835 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.357807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:34.359725 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:26:34.359698    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1022a9b1_f826_4ade_9d9c_26329fee0006.slice/crio-16c65ea1891de2b4de340a2f1426fd6fa544835015dfeed33d7c2655c79f6cac WatchSource:0}: Error finding container 16c65ea1891de2b4de340a2f1426fd6fa544835015dfeed33d7c2655c79f6cac: Status 404 returned error can't find the container with id 16c65ea1891de2b4de340a2f1426fd6fa544835015dfeed33d7c2655c79f6cac
Jul 23 00:26:34.361147 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:34.361125    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:26:35.334498 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:35.334462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" event={"ID":"1022a9b1-f826-4ade-9d9c-26329fee0006","Type":"ContainerStarted","Data":"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"}
Jul 23 00:26:35.334899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:35.334511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" event={"ID":"1022a9b1-f826-4ade-9d9c-26329fee0006","Type":"ContainerStarted","Data":"16c65ea1891de2b4de340a2f1426fd6fa544835015dfeed33d7c2655c79f6cac"}
Jul 23 00:26:35.334899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:35.334619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:35.351709 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:35.351623    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" podStartSLOduration=1.790203944 podStartE2EDuration="2.351608423s" totalImagesPullingTime="561.404479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:35.351043119 +0000 UTC m=+1130.204612793" watchObservedRunningTime="2026-07-23 00:26:35.351608423 +0000 UTC m=+1130.205178097"
Jul 23 00:26:46.340960 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:46.340925    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:50.992142 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:50.992105    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:50.992539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:50.992322    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" podUID="1022a9b1-f826-4ade-9d9c-26329fee0006" containerName="manager" containerID="cri-o://5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232" gracePeriod=30
Jul 23 00:26:51.233052 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.233028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:51.378378 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.378292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln2p8\" (UniqueName: \"kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8\") pod \"1022a9b1-f826-4ade-9d9c-26329fee0006\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") "
Jul 23 00:26:51.378378 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.378351    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert\") pod \"1022a9b1-f826-4ade-9d9c-26329fee0006\" (UID: \"1022a9b1-f826-4ade-9d9c-26329fee0006\") "
Jul 23 00:26:51.380457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.380422    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1022a9b1-f826-4ade-9d9c-26329fee0006" (UID: "1022a9b1-f826-4ade-9d9c-26329fee0006"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:26:51.380571 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.380480    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8" (OuterVolumeSpecName: "kube-api-access-ln2p8") pod "1022a9b1-f826-4ade-9d9c-26329fee0006" (UID: "1022a9b1-f826-4ade-9d9c-26329fee0006"). InnerVolumeSpecName "kube-api-access-ln2p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:26:51.418332 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.418297    2565 generic.go:356] "Generic (PLEG): container finished" podID="1022a9b1-f826-4ade-9d9c-26329fee0006" containerID="5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232" exitCode=0
Jul 23 00:26:51.418501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.418343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-ft8j8"
Jul 23 00:26:51.418501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.418385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" event={"ID":"1022a9b1-f826-4ade-9d9c-26329fee0006","Type":"ContainerDied","Data":"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"}
Jul 23 00:26:51.418501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.418443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-ft8j8" event={"ID":"1022a9b1-f826-4ade-9d9c-26329fee0006","Type":"ContainerDied","Data":"16c65ea1891de2b4de340a2f1426fd6fa544835015dfeed33d7c2655c79f6cac"}
Jul 23 00:26:51.418501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.418472    2565 scope.go:117] "RemoveContainer" containerID="5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"
Jul 23 00:26:51.427780 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.427762    2565 scope.go:117] "RemoveContainer" containerID="5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"
Jul 23 00:26:51.428070 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:26:51.428051    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232\": container with ID starting with 5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232 not found: ID does not exist" containerID="5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"
Jul 23 00:26:51.428157 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.428079    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232"} err="failed to get container status \"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232\": rpc error: code = NotFound desc = could not find container \"5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232\": container with ID starting with 5823592095aae85a0a5ed6b69efdac097d100123b9ab0006d18b43fcb62ba232 not found: ID does not exist"
Jul 23 00:26:51.440211 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.440191    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:51.444095 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.444072    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-ft8j8"]
Jul 23 00:26:51.479554 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.479528    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2p8\" (UniqueName: \"kubernetes.io/projected/1022a9b1-f826-4ade-9d9c-26329fee0006-kube-api-access-ln2p8\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:26:51.479554 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.479552    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1022a9b1-f826-4ade-9d9c-26329fee0006-webhook-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:26:51.725862 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:51.725781    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1022a9b1-f826-4ade-9d9c-26329fee0006" path="/var/lib/kubelet/pods/1022a9b1-f826-4ade-9d9c-26329fee0006/volumes"
Jul 23 00:26:52.252166 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.252133    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fccc86db5-2ccj2"]
Jul 23 00:26:52.252677 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.252658    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1022a9b1-f826-4ade-9d9c-26329fee0006" containerName="manager"
Jul 23 00:26:52.252736 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.252681    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1022a9b1-f826-4ade-9d9c-26329fee0006" containerName="manager"
Jul 23 00:26:52.252819 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.252808    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1022a9b1-f826-4ade-9d9c-26329fee0006" containerName="manager"
Jul 23 00:26:52.257287 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.257266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.260120 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.260095    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 00:26:52.261323 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.261304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 00:26:52.261455 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.261381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4t6cw\""
Jul 23 00:26:52.264779 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.264442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-2ccj2"]
Jul 23 00:26:52.387464 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.387433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk542\" (UniqueName: \"kubernetes.io/projected/0d6840e7-577e-4d3c-a391-81e4d76b85d3-kube-api-access-sk542\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.387609 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.387553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6840e7-577e-4d3c-a391-81e4d76b85d3-webhook-cert\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.488281 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.488246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk542\" (UniqueName: \"kubernetes.io/projected/0d6840e7-577e-4d3c-a391-81e4d76b85d3-kube-api-access-sk542\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.488436 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.488352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6840e7-577e-4d3c-a391-81e4d76b85d3-webhook-cert\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.490717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.490675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d6840e7-577e-4d3c-a391-81e4d76b85d3-webhook-cert\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.496000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.495956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk542\" (UniqueName: \"kubernetes.io/projected/0d6840e7-577e-4d3c-a391-81e4d76b85d3-kube-api-access-sk542\") pod \"maas-controller-5fccc86db5-2ccj2\" (UID: \"0d6840e7-577e-4d3c-a391-81e4d76b85d3\") " pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.569465 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.569391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:52.703376 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:52.703345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fccc86db5-2ccj2"]
Jul 23 00:26:52.706517 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:26:52.706491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6840e7_577e_4d3c_a391_81e4d76b85d3.slice/crio-9be2e84f6cefcc9297cd2f9fbc976333e7376ac6047664ef9eadd8822bf62a01 WatchSource:0}: Error finding container 9be2e84f6cefcc9297cd2f9fbc976333e7376ac6047664ef9eadd8822bf62a01: Status 404 returned error can't find the container with id 9be2e84f6cefcc9297cd2f9fbc976333e7376ac6047664ef9eadd8822bf62a01
Jul 23 00:26:53.431616 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:53.431528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-2ccj2" event={"ID":"0d6840e7-577e-4d3c-a391-81e4d76b85d3","Type":"ContainerStarted","Data":"c77021a31b7f6b09a309ba82c1464989fcc269f5ec2e65c71ddf644166f3c6e0"}
Jul 23 00:26:53.431616 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:53.431579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fccc86db5-2ccj2" event={"ID":"0d6840e7-577e-4d3c-a391-81e4d76b85d3","Type":"ContainerStarted","Data":"9be2e84f6cefcc9297cd2f9fbc976333e7376ac6047664ef9eadd8822bf62a01"}
Jul 23 00:26:53.432056 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:53.431664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:26:53.450691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:26:53.450646    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fccc86db5-2ccj2" podStartSLOduration=1.104965925 podStartE2EDuration="1.45062912s" totalImagesPullingTime="345.663195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:26:53.449466828 +0000 UTC m=+1148.303036502" watchObservedRunningTime="2026-07-23 00:26:53.45062912 +0000 UTC m=+1148.304198810"
Jul 23 00:27:04.439345 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:27:04.439306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fccc86db5-2ccj2"
Jul 23 00:27:17.230155 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:27:17.230120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:27:24.826586 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:27:24.826554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:27:56.932189 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:27:56.932137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:06.429294 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:06.429254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:14.935640 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:14.935599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:24.520372 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:24.520336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:33.629354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:33.629315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:45.023549 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:45.023469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:28:56.136390 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:28:56.136348    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:29:06.029497 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:29:06.029466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:29:28.032206 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:29:28.032165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:29:37.334312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:29:37.334273    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:00.025629 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:00.025583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:09.826078 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:09.825989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:25.325934 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:25.325890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:34.430819 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:34.430784    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:43.726075 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:43.726038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:30:53.225848 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:30:53.225812    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:31:02.427774 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:02.427727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:31:29.872587 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:29.872555    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:31:29.873151 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:29.872752    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" podUID="16174b30-12f9-45cd-a1d4-88064961f598" containerName="manager" containerID="cri-o://e27e3b6c499357c946cfbaa5b6f6412ce33b7668b1661596d10e94e28b31d879" gracePeriod=10
Jul 23 00:31:32.709514 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.709478    2565 generic.go:356] "Generic (PLEG): container finished" podID="16174b30-12f9-45cd-a1d4-88064961f598" containerID="e27e3b6c499357c946cfbaa5b6f6412ce33b7668b1661596d10e94e28b31d879" exitCode=0
Jul 23 00:31:32.709935 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.709555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" event={"ID":"16174b30-12f9-45cd-a1d4-88064961f598","Type":"ContainerDied","Data":"e27e3b6c499357c946cfbaa5b6f6412ce33b7668b1661596d10e94e28b31d879"}
Jul 23 00:31:32.823807 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.823781    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:31:32.883673 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.883644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffkvd\" (UniqueName: \"kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd\") pod \"16174b30-12f9-45cd-a1d4-88064961f598\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") "
Jul 23 00:31:32.883814 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.883684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume\") pod \"16174b30-12f9-45cd-a1d4-88064961f598\" (UID: \"16174b30-12f9-45cd-a1d4-88064961f598\") "
Jul 23 00:31:32.884035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.884015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "16174b30-12f9-45cd-a1d4-88064961f598" (UID: "16174b30-12f9-45cd-a1d4-88064961f598"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:31:32.885626 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.885605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd" (OuterVolumeSpecName: "kube-api-access-ffkvd") pod "16174b30-12f9-45cd-a1d4-88064961f598" (UID: "16174b30-12f9-45cd-a1d4-88064961f598"). InnerVolumeSpecName "kube-api-access-ffkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:31:32.984521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.984453    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkvd\" (UniqueName: \"kubernetes.io/projected/16174b30-12f9-45cd-a1d4-88064961f598-kube-api-access-ffkvd\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:31:32.984521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:32.984477    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16174b30-12f9-45cd-a1d4-88064961f598-extensions-socket-volume\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:31:33.714833 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:33.714805    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"
Jul 23 00:31:33.715299 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:33.714799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9" event={"ID":"16174b30-12f9-45cd-a1d4-88064961f598","Type":"ContainerDied","Data":"d026a13d86f25f47b29805f40538674f2701fd32c32f796577f6a4c3820e3fb3"}
Jul 23 00:31:33.715299 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:33.714945    2565 scope.go:117] "RemoveContainer" containerID="e27e3b6c499357c946cfbaa5b6f6412ce33b7668b1661596d10e94e28b31d879"
Jul 23 00:31:33.740600 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:33.740576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:31:33.747454 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:33.747430    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x48k9"]
Jul 23 00:31:35.726072 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:31:35.726039    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16174b30-12f9-45cd-a1d4-88064961f598" path="/var/lib/kubelet/pods/16174b30-12f9-45cd-a1d4-88064961f598/volumes"
Jul 23 00:32:31.045771 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.045740    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"]
Jul 23 00:32:31.048068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.046091    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16174b30-12f9-45cd-a1d4-88064961f598" containerName="manager"
Jul 23 00:32:31.048068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.046104    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="16174b30-12f9-45cd-a1d4-88064961f598" containerName="manager"
Jul 23 00:32:31.048068 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.046162    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="16174b30-12f9-45cd-a1d4-88064961f598" containerName="manager"
Jul 23 00:32:31.049030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.049015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.051846 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.051827    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rx7pw\""
Jul 23 00:32:31.061247 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.061225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"]
Jul 23 00:32:31.185286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.185249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8ft\" (UniqueName: \"kubernetes.io/projected/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-kube-api-access-kl8ft\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.185286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.185294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.285899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.285874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8ft\" (UniqueName: \"kubernetes.io/projected/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-kube-api-access-kl8ft\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.286054 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.285911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.286296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.286278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.293822 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.293801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8ft\" (UniqueName: \"kubernetes.io/projected/a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b-kube-api-access-kl8ft\") pod \"kuadrant-operator-controller-manager-86878c769f-jgq8m\" (UID: \"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.358775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.358714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.490924 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.490902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"]
Jul 23 00:32:31.492961 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:32:31.492919    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb9da3_9f3b_49cd_8b22_b93b7d5d1c6b.slice/crio-b2f272b9d5c0e5b6806193f3eeb28c5880fbe2f0ebed0c5cc21fdb3eb5171abc WatchSource:0}: Error finding container b2f272b9d5c0e5b6806193f3eeb28c5880fbe2f0ebed0c5cc21fdb3eb5171abc: Status 404 returned error can't find the container with id b2f272b9d5c0e5b6806193f3eeb28c5880fbe2f0ebed0c5cc21fdb3eb5171abc
Jul 23 00:32:31.974597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.974561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m" event={"ID":"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b","Type":"ContainerStarted","Data":"b0ded2145d7fd1584ea6ebbe5c9aa6e4158f814997f7c14fcaac1874426f2c56"}
Jul 23 00:32:31.974754 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.974622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m" event={"ID":"a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b","Type":"ContainerStarted","Data":"b2f272b9d5c0e5b6806193f3eeb28c5880fbe2f0ebed0c5cc21fdb3eb5171abc"}
Jul 23 00:32:31.974754 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.974645    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:31.995293 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:31.995244    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m" podStartSLOduration=0.995231944 podStartE2EDuration="995.231944ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:32:31.995058321 +0000 UTC m=+1486.848627993" watchObservedRunningTime="2026-07-23 00:32:31.995231944 +0000 UTC m=+1486.848801616"
Jul 23 00:32:42.981877 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:42.981844    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jgq8m"
Jul 23 00:32:50.931704 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:32:50.931668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:01.924858 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:01.924818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:11.328354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:11.328274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:23.126815 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:23.126776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:34.228093 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:34.228055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:44.725928 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:44.725890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:33:54.130667 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:33:54.130629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:34:05.425843 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:34:05.425808    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:34:16.319646 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:34:16.319614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:34:25.622035 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:34:25.622002    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:34:37.232805 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:34:37.232773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:35:02.722383 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:02.722351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:35:10.520209 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:10.520173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:35:32.535984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.535924    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:35:32.539401 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.539385    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.542104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.542085    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c35c73c7-kserve-self-signed-certs\""
Jul 23 00:35:32.554775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.554753    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:35:32.705099 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.705277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rwf\" (UniqueName: \"kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.705277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.705277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.705277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.705457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.705304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806143 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806143 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806344 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806344 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806344 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806344 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rwf\" (UniqueName: \"kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806651 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806786 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.806851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.806791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.808507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.808482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.808651 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.808632    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.813995 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.813954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rwf\" (UniqueName: \"kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf\") pod \"e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") " pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.848950 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.848923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:32.981488 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:32.981465    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:35:32.982901 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:35:32.982875    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb97261af_d8e6_4250_b3d2_b6c0b9cfcff5.slice/crio-93da9e73635bd95c6ad91e582e4decc6756c7f4e23f8b9d3a27d65649122e23f WatchSource:0}: Error finding container 93da9e73635bd95c6ad91e582e4decc6756c7f4e23f8b9d3a27d65649122e23f: Status 404 returned error can't find the container with id 93da9e73635bd95c6ad91e582e4decc6756c7f4e23f8b9d3a27d65649122e23f
Jul 23 00:35:33.817148 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:33.817118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" event={"ID":"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5","Type":"ContainerStarted","Data":"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"}
Jul 23 00:35:33.817526 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:33.817157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" event={"ID":"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5","Type":"ContainerStarted","Data":"93da9e73635bd95c6ad91e582e4decc6756c7f4e23f8b9d3a27d65649122e23f"}
Jul 23 00:35:33.834487 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:33.834443    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" podStartSLOduration=1.834429155 podStartE2EDuration="1.834429155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:35:33.833640984 +0000 UTC m=+1668.687210661" watchObservedRunningTime="2026-07-23 00:35:33.834429155 +0000 UTC m=+1668.687998825"
Jul 23 00:35:35.818676 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:35.818639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:35:42.850067 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:42.849960    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:42.850067 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:42.850030    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:42.862932 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:42.862909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:42.874047 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:42.874024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:35:53.617263 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:35:53.617227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:36:23.510018 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:23.509912    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:36:26.322027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:26.321989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:36:38.531449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.531408    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" podUID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" containerName="main" containerID="cri-o://c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a" gracePeriod=45
Jul 23 00:36:38.783879 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.783816    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:36:38.871525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871478    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871611    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871687 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871665    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871693    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27rwf\" (UniqueName: \"kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf\") pod \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\" (UID: \"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5\") "
Jul 23 00:36:38.871872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871788    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home" (OuterVolumeSpecName: "home") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:36:38.872048 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.871997    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:38.872048 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.872023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:36:38.872311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.872267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:36:38.873883 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.873862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm" (OuterVolumeSpecName: "dshm") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:36:38.874046 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.874028    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:36:38.874178 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.874161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf" (OuterVolumeSpecName: "kube-api-access-27rwf") pod "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" (UID: "b97261af-d8e6-4250-b3d2-b6c0b9cfcff5"). InnerVolumeSpecName "kube-api-access-27rwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:36:38.973079 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.973039    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:38.973079 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.973078    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:38.973311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.973093    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:38.973311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.973105    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:38.973311 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:38.973118    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rwf\" (UniqueName: \"kubernetes.io/projected/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5-kube-api-access-27rwf\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:36:39.120527 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.120445    2565 generic.go:356] "Generic (PLEG): container finished" podID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" containerID="c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a" exitCode=0
Jul 23 00:36:39.120527 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.120492    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"
Jul 23 00:36:39.120703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.120527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" event={"ID":"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5","Type":"ContainerDied","Data":"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"}
Jul 23 00:36:39.120703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.120573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg" event={"ID":"b97261af-d8e6-4250-b3d2-b6c0b9cfcff5","Type":"ContainerDied","Data":"93da9e73635bd95c6ad91e582e4decc6756c7f4e23f8b9d3a27d65649122e23f"}
Jul 23 00:36:39.120703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.120592    2565 scope.go:117] "RemoveContainer" containerID="c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"
Jul 23 00:36:39.129395 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.129377    2565 scope.go:117] "RemoveContainer" containerID="c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"
Jul 23 00:36:39.129656 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:36:39.129637    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a\": container with ID starting with c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a not found: ID does not exist" containerID="c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"
Jul 23 00:36:39.129708 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.129665    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a"} err="failed to get container status \"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a\": rpc error: code = NotFound desc = could not find container \"c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a\": container with ID starting with c9c4c6e62eb670f815a9b97250a98663216f61fa2c452136378da5f6ef3a0d6a not found: ID does not exist"
Jul 23 00:36:39.141252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.141233    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:36:39.144851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.144834    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c35c73c7-kserve-5564f76655-9ngmg"]
Jul 23 00:36:39.725979 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:39.725930    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" path="/var/lib/kubelet/pods/b97261af-d8e6-4250-b3d2-b6c0b9cfcff5/volumes"
Jul 23 00:36:40.226114 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:40.226081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:36:48.121444 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:36:48.121409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:02.933526 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:02.933489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:13.130164 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:13.130123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:22.427519 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:22.427482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:31.727458 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:31.727426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:42.422745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:42.422672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:37:51.717946 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:37:51.717908    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:05.241910 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:05.241875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:14.329302 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:14.329264    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:23.923591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:23.923553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:34.421393 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:34.421354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:43.831136 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:43.831099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:38:54.734342 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:38:54.734308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:39:10.926136 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:39:10.926043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:39:23.327364 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:39:23.327332    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:39:36.320257 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:39:36.320219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:39:45.122345 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:39:45.122312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:40:06.228944 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:06.228903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:40:22.319515 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:22.319479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:40:40.519293 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.519190    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"]
Jul 23 00:40:40.519762 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.519550    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" containerName="main"
Jul 23 00:40:40.519762 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.519560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" containerName="main"
Jul 23 00:40:40.519762 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.519628    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97261af-d8e6-4250-b3d2-b6c0b9cfcff5" containerName="main"
Jul 23 00:40:40.522588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.522566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:40.525224 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.525205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:40:40.525338 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.525244    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:40:40.529097 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.529073    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"]
Jul 23 00:40:40.663248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.663216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj\" (UID: \"42c2dd8d-2307-4a6b-939d-fe3637775ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:40.764166 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.764137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj\" (UID: \"42c2dd8d-2307-4a6b-939d-fe3637775ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:40.764740 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.764721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj\" (UID: \"42c2dd8d-2307-4a6b-939d-fe3637775ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:40.833839 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.833759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:40.969913 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.969887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"]
Jul 23 00:40:40.971473 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:40:40.971449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c2dd8d_2307_4a6b_939d_fe3637775ab4.slice/crio-875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab WatchSource:0}: Error finding container 875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab: Status 404 returned error can't find the container with id 875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab
Jul 23 00:40:40.973411 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:40.973392    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:40:41.244558 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:41.244530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj" event={"ID":"42c2dd8d-2307-4a6b-939d-fe3637775ab4","Type":"ContainerStarted","Data":"875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab"}
Jul 23 00:40:55.313836 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:55.313800    2565 generic.go:356] "Generic (PLEG): container finished" podID="42c2dd8d-2307-4a6b-939d-fe3637775ab4" containerID="95edcef4ab9f1f0e64b74b6e82facff5432a7b6a2ea2e4c786c5aa214d93c34b" exitCode=0
Jul 23 00:40:55.314267 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:55.313885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj" event={"ID":"42c2dd8d-2307-4a6b-939d-fe3637775ab4","Type":"ContainerDied","Data":"95edcef4ab9f1f0e64b74b6e82facff5432a7b6a2ea2e4c786c5aa214d93c34b"}
Jul 23 00:40:56.457070 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:56.457040    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:56.497226 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:56.497194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca\") pod \"42c2dd8d-2307-4a6b-939d-fe3637775ab4\" (UID: \"42c2dd8d-2307-4a6b-939d-fe3637775ab4\") "
Jul 23 00:40:56.497534 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:56.497511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42c2dd8d-2307-4a6b-939d-fe3637775ab4" (UID: "42c2dd8d-2307-4a6b-939d-fe3637775ab4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:40:56.598581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:56.598503    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2dd8d-2307-4a6b-939d-fe3637775ab4-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:40:57.324129 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:57.324103    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"
Jul 23 00:40:57.324307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:57.324138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj" event={"ID":"42c2dd8d-2307-4a6b-939d-fe3637775ab4","Type":"ContainerDied","Data":"875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab"}
Jul 23 00:40:57.324307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:40:57.324187    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="875dc7e469b0708b1c57025470ebaddccf03790ce522f07907a80b945beb4dab"
Jul 23 00:41:03.726304 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:03.726265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:41:05.595821 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:05.595790    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"]
Jul 23 00:41:05.598149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:05.598124    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eadb28c-rrzsj"]
Jul 23 00:41:05.725694 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:05.725648    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c2dd8d-2307-4a6b-939d-fe3637775ab4" path="/var/lib/kubelet/pods/42c2dd8d-2307-4a6b-939d-fe3637775ab4/volumes"
Jul 23 00:41:06.779055 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.779025    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:06.779445 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.779413    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c2dd8d-2307-4a6b-939d-fe3637775ab4" containerName="revoke-keys"
Jul 23 00:41:06.779445 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.779425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c2dd8d-2307-4a6b-939d-fe3637775ab4" containerName="revoke-keys"
Jul 23 00:41:06.779560 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.779493    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c2dd8d-2307-4a6b-939d-fe3637775ab4" containerName="revoke-keys"
Jul 23 00:41:06.782666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.782646    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.785720 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.785690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-72954db1-gw-openshift-default-dockercfg-4cq7z\""
Jul 23 00:41:06.796259 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.796237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:06.885891 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.885860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.885891 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.885891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.885932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tbz\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886384 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.886384 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.886212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.986679 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.986645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.986857 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.986703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.986857 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.986732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.986857 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.986748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.986960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tbz\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987156    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987296 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987724 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987724 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.987840 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.987820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.988142 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.988119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.989231 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.989209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.989231 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.989227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.995023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.995004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:06.995125 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:06.995094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tbz\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz\") pod \"e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") " pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:07.093327 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.093255    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:07.228329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.228305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:07.229729 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:41:07.229694    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683bae69_914c_4dce_9e76_9a131630e423.slice/crio-13ff03cf006a4dcd2af8a2272cb159faddb28c17f98419d34b4687b4865ca5f9 WatchSource:0}: Error finding container 13ff03cf006a4dcd2af8a2272cb159faddb28c17f98419d34b4687b4865ca5f9: Status 404 returned error can't find the container with id 13ff03cf006a4dcd2af8a2272cb159faddb28c17f98419d34b4687b4865ca5f9
Jul 23 00:41:07.231906 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.231875    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:07.231991 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.231936    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:07.231991 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.231964    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:07.380336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.380254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerStarted","Data":"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"}
Jul 23 00:41:07.380336 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.380332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerStarted","Data":"13ff03cf006a4dcd2af8a2272cb159faddb28c17f98419d34b4687b4865ca5f9"}
Jul 23 00:41:07.401755 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:07.401697    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podStartSLOduration=1.40168172 podStartE2EDuration="1.40168172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:41:07.399897175 +0000 UTC m=+2002.253466848" watchObservedRunningTime="2026-07-23 00:41:07.40168172 +0000 UTC m=+2002.255251473"
Jul 23 00:41:08.093768 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:08.093734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:08.098705 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:08.098683    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:08.385877 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:08.385799    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:08.386683 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:08.386664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:15.927489 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:15.927453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:41:23.537138 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:23.537101    2565 patch_prober.go:28] interesting pod/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": read tcp 10.132.0.2:47440->10.132.0.46:15021: read: connection reset by peer" start-of-body=
Jul 23 00:41:23.537590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:23.537173    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": read tcp 10.132.0.2:47440->10.132.0.46:15021: read: connection reset by peer"
Jul 23 00:41:24.459513 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.459480    2565 generic.go:356] "Generic (PLEG): container finished" podID="683bae69-914c-4dce-9e76-9a131630e423" containerID="a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28" exitCode=0
Jul 23 00:41:24.459731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.459555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerDied","Data":"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"}
Jul 23 00:41:24.459995 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.459961    2565 scope.go:117] "RemoveContainer" containerID="a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"
Jul 23 00:41:24.461020 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.460988    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:24.461096 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.461058    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:24.461096 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:24.461088    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:41:25.467255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:25.466550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerStarted","Data":"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"}
Jul 23 00:41:26.093454 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:26.093420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:26.093661 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:26.093639    2565 patch_prober.go:28] interesting pod/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 23 00:41:26.093709 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:26.093690    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 23 00:41:27.094117 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:27.094077    2565 patch_prober.go:28] interesting pod/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 23 00:41:27.094507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:27.094142    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 23 00:41:28.093985 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:28.093939    2565 patch_prober.go:28] interesting pod/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 23 00:41:28.094179 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:28.094032    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 23 00:41:29.094576 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:29.094536    2565 patch_prober.go:28] interesting pod/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 23 00:41:29.095006 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:29.094594    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 23 00:41:30.096828 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:30.096800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:30.097216 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:30.097075    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:30.097830 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:30.097811    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:47.225044 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:47.225008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:41:49.142652 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:49.142620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:49.143133 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:49.142855    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy" containerID="cri-o://64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1" gracePeriod=30
Jul 23 00:41:54.403449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.403391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:54.499926 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.499898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.499944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2tbz\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500033    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500145    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy\") pod \"683bae69-914c-4dce-9e76-9a131630e423\" (UID: \"683bae69-914c-4dce-9e76-9a131630e423\") "
Jul 23 00:41:54.500508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:41:54.500508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:41:54.500508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500485    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.500508 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500503    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-credential-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.500771 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:41:54.500771 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:41:54.500889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.500868    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data" (OuterVolumeSpecName: "istio-data") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:41:54.502364 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.502329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:41:54.502469 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.502421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token" (OuterVolumeSpecName: "istio-token") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:41:54.502588 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.502566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 00:41:54.502633 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.502607    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz" (OuterVolumeSpecName: "kube-api-access-s2tbz") pod "683bae69-914c-4dce-9e76-9a131630e423" (UID: "683bae69-914c-4dce-9e76-9a131630e423"). InnerVolumeSpecName "kube-api-access-s2tbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:41:54.601225 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601190    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-data\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601225 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601218    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/683bae69-914c-4dce-9e76-9a131630e423-istiod-ca-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601225 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601228    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/683bae69-914c-4dce-9e76-9a131630e423-istio-podinfo\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601431 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601235    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-workload-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601431 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601244    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/683bae69-914c-4dce-9e76-9a131630e423-istio-envoy\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601431 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601259    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-istio-token\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.601431 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.601275    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2tbz\" (UniqueName: \"kubernetes.io/projected/683bae69-914c-4dce-9e76-9a131630e423-kube-api-access-s2tbz\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:41:54.630584 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.630555    2565 generic.go:356] "Generic (PLEG): container finished" podID="683bae69-914c-4dce-9e76-9a131630e423" containerID="64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1" exitCode=0
Jul 23 00:41:54.630703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.630604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"
Jul 23 00:41:54.630703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.630609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerDied","Data":"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"}
Jul 23 00:41:54.630703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.630651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft" event={"ID":"683bae69-914c-4dce-9e76-9a131630e423","Type":"ContainerDied","Data":"13ff03cf006a4dcd2af8a2272cb159faddb28c17f98419d34b4687b4865ca5f9"}
Jul 23 00:41:54.630703 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.630672    2565 scope.go:117] "RemoveContainer" containerID="64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"
Jul 23 00:41:54.641092 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.641067    2565 scope.go:117] "RemoveContainer" containerID="a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"
Jul 23 00:41:54.648525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.648508    2565 scope.go:117] "RemoveContainer" containerID="64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"
Jul 23 00:41:54.648728 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:41:54.648712    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1\": container with ID starting with 64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1 not found: ID does not exist" containerID="64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"
Jul 23 00:41:54.648775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.648734    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1"} err="failed to get container status \"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1\": rpc error: code = NotFound desc = could not find container \"64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1\": container with ID starting with 64dd4599ae67e50141af334205b534c1321eee0595d02eea30adfb0768190df1 not found: ID does not exist"
Jul 23 00:41:54.648775 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.648750    2565 scope.go:117] "RemoveContainer" containerID="a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"
Jul 23 00:41:54.648943 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:41:54.648926    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28\": container with ID starting with a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28 not found: ID does not exist" containerID="a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"
Jul 23 00:41:54.649002 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.648949    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28"} err="failed to get container status \"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28\": rpc error: code = NotFound desc = could not find container \"a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28\": container with ID starting with a0919a67018da60b3fc2a156d951cc5605287ffdfc607344e602954c95a9aa28 not found: ID does not exist"
Jul 23 00:41:54.653157 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.653135    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:54.656939 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:54.656892    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-72954db1-gw-openshift-default-55c867c4c-jbgft"]
Jul 23 00:41:55.726031 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:55.725997    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="683bae69-914c-4dce-9e76-9a131630e423" path="/var/lib/kubelet/pods/683bae69-914c-4dce-9e76-9a131630e423/volumes"
Jul 23 00:41:59.321475 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:41:59.321434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:42:18.395080 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.395002    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"]
Jul 23 00:42:18.395448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.395336    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:42:18.395448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.395347    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:42:18.395448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.395414    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:42:18.399721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.399701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:18.402408 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.402389    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:42:18.402509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.402431    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:42:18.406097 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.406072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"]
Jul 23 00:42:18.486924 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.486887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2\" (UID: \"6a63d103-1e34-4ad2-b8cd-098f9a116e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:18.588039 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.588005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2\" (UID: \"6a63d103-1e34-4ad2-b8cd-098f9a116e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:18.588691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.588667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2\" (UID: \"6a63d103-1e34-4ad2-b8cd-098f9a116e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:18.710807 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:18.710728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:19.047430 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:19.047402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"]
Jul 23 00:42:19.049072 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:42:19.049047    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a63d103_1e34_4ad2_b8cd_098f9a116e77.slice/crio-7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2 WatchSource:0}: Error finding container 7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2: Status 404 returned error can't find the container with id 7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2
Jul 23 00:42:19.750315 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:19.750288    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a63d103-1e34-4ad2-b8cd-098f9a116e77" containerID="b2b9f7fd29c683da0ca5a67c2bd69d539e780eef3f57bdf3e31dd7e1c175104b" exitCode=0
Jul 23 00:42:19.750608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:19.750374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2" event={"ID":"6a63d103-1e34-4ad2-b8cd-098f9a116e77","Type":"ContainerDied","Data":"b2b9f7fd29c683da0ca5a67c2bd69d539e780eef3f57bdf3e31dd7e1c175104b"}
Jul 23 00:42:19.750608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:19.750432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2" event={"ID":"6a63d103-1e34-4ad2-b8cd-098f9a116e77","Type":"ContainerStarted","Data":"7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2"}
Jul 23 00:42:20.890114 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:20.890091    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:21.008937 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.008866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca\") pod \"6a63d103-1e34-4ad2-b8cd-098f9a116e77\" (UID: \"6a63d103-1e34-4ad2-b8cd-098f9a116e77\") "
Jul 23 00:42:21.009211 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.009190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a63d103-1e34-4ad2-b8cd-098f9a116e77" (UID: "6a63d103-1e34-4ad2-b8cd-098f9a116e77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:42:21.109594 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.109568    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a63d103-1e34-4ad2-b8cd-098f9a116e77-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:42:21.762831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.762802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2" event={"ID":"6a63d103-1e34-4ad2-b8cd-098f9a116e77","Type":"ContainerDied","Data":"7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2"}
Jul 23 00:42:21.762985 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.762837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"
Jul 23 00:42:21.762985 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:21.762847    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb18fdfe7e7120693fede7bd070ea600ad35d2365f5f6741cf4306bc071eff2"
Jul 23 00:42:31.825386 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:31.825350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:42:33.474269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:33.474225    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"]
Jul 23 00:42:33.478358 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:33.478331    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d0be4a18-6jgl2"]
Jul 23 00:42:33.726672 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:33.726588    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a63d103-1e34-4ad2-b8cd-098f9a116e77" path="/var/lib/kubelet/pods/6a63d103-1e34-4ad2-b8cd-098f9a116e77/volumes"
Jul 23 00:42:44.629835 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:42:44.629800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:43:02.649855 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.649820    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"]
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650191    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a63d103-1e34-4ad2-b8cd-098f9a116e77" containerName="revoke-keys"
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650203    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a63d103-1e34-4ad2-b8cd-098f9a116e77" containerName="revoke-keys"
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650221    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650226    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650287    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="683bae69-914c-4dce-9e76-9a131630e423" containerName="istio-proxy"
Jul 23 00:43:02.650348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.650297    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a63d103-1e34-4ad2-b8cd-098f9a116e77" containerName="revoke-keys"
Jul 23 00:43:02.653141 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.653126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:02.655865 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.655842    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:43:02.656036 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.655842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:43:02.659917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.659898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"]
Jul 23 00:43:02.775745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.775709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm\" (UID: \"00a2b08d-fa34-45c5-a08c-ecb091b52a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:02.876503 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.876466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm\" (UID: \"00a2b08d-fa34-45c5-a08c-ecb091b52a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:02.877092 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.877072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm\" (UID: \"00a2b08d-fa34-45c5-a08c-ecb091b52a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:02.964261 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:02.964173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:03.095457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:03.095334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"]
Jul 23 00:43:03.097701 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:43:03.097661    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a2b08d_fa34_45c5_a08c_ecb091b52a38.slice/crio-6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0 WatchSource:0}: Error finding container 6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0: Status 404 returned error can't find the container with id 6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0
Jul 23 00:43:03.967121 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:03.967086    2565 generic.go:356] "Generic (PLEG): container finished" podID="00a2b08d-fa34-45c5-a08c-ecb091b52a38" containerID="557288dca0cd7901c2207f7649bb0698505368bd46c9d7fad481a3a7212b23f8" exitCode=0
Jul 23 00:43:03.967478 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:03.967157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm" event={"ID":"00a2b08d-fa34-45c5-a08c-ecb091b52a38","Type":"ContainerDied","Data":"557288dca0cd7901c2207f7649bb0698505368bd46c9d7fad481a3a7212b23f8"}
Jul 23 00:43:03.967478 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:03.967195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm" event={"ID":"00a2b08d-fa34-45c5-a08c-ecb091b52a38","Type":"ContainerStarted","Data":"6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0"}
Jul 23 00:43:05.098638 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.098616    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:05.196114 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.196085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca\") pod \"00a2b08d-fa34-45c5-a08c-ecb091b52a38\" (UID: \"00a2b08d-fa34-45c5-a08c-ecb091b52a38\") "
Jul 23 00:43:05.196417 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.196396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00a2b08d-fa34-45c5-a08c-ecb091b52a38" (UID: "00a2b08d-fa34-45c5-a08c-ecb091b52a38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:43:05.296717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.296688    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a2b08d-fa34-45c5-a08c-ecb091b52a38-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:43:05.982507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.982418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm" event={"ID":"00a2b08d-fa34-45c5-a08c-ecb091b52a38","Type":"ContainerDied","Data":"6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0"}
Jul 23 00:43:05.982507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.982493    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6381f9591a1d3590fcfe5df20c0020263cb3a9991422ebc06eb3fe729e676ca0"
Jul 23 00:43:05.982507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:05.982436    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"
Jul 23 00:43:15.822591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:15.822548    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:43:17.732052 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:17.732023    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"]
Jul 23 00:43:17.732052 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:17.732058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3fad28dc-6bjcm"]
Jul 23 00:43:19.725655 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:43:19.725620    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a2b08d-fa34-45c5-a08c-ecb091b52a38" path="/var/lib/kubelet/pods/00a2b08d-fa34-45c5-a08c-ecb091b52a38/volumes"
Jul 23 00:44:47.231692 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:44:47.231657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:44:57.021773 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:44:57.021740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:45:09.329412 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:09.329327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:45:15.834093 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.834055    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"]
Jul 23 00:45:15.834491 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.834447    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a2b08d-fa34-45c5-a08c-ecb091b52a38" containerName="revoke-keys"
Jul 23 00:45:15.834491 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.834460    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a2b08d-fa34-45c5-a08c-ecb091b52a38" containerName="revoke-keys"
Jul 23 00:45:15.834572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.834526    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a2b08d-fa34-45c5-a08c-ecb091b52a38" containerName="revoke-keys"
Jul 23 00:45:15.837768 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.837745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:15.840274 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.840250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:45:15.840489 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.840475    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:45:15.844122 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:15.844101    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"]
Jul 23 00:45:16.000355 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.000322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c65690b4-p4twz\" (UID: \"feeb9da4-a330-4699-a1a4-c897972b3937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:16.101586 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.101497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c65690b4-p4twz\" (UID: \"feeb9da4-a330-4699-a1a4-c897972b3937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:16.102104 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.102084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c65690b4-p4twz\" (UID: \"feeb9da4-a330-4699-a1a4-c897972b3937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:16.149717 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.149686    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:16.291818 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.291793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"]
Jul 23 00:45:16.294356 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:45:16.294318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeeb9da4_a330_4699_a1a4_c897972b3937.slice/crio-d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7 WatchSource:0}: Error finding container d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7: Status 404 returned error can't find the container with id d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7
Jul 23 00:45:16.611756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.611722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz" event={"ID":"feeb9da4-a330-4699-a1a4-c897972b3937","Type":"ContainerStarted","Data":"9c139a264654779197fa152ab654944f9c0e183673edc90593295e6dfe7fbbab"}
Jul 23 00:45:16.611903 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.611784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz" event={"ID":"feeb9da4-a330-4699-a1a4-c897972b3937","Type":"ContainerStarted","Data":"d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7"}
Jul 23 00:45:16.628521 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:16.628478    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz" podStartSLOduration=1.628462817 podStartE2EDuration="1.628462817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:45:16.627206952 +0000 UTC m=+2251.480776625" watchObservedRunningTime="2026-07-23 00:45:16.628462817 +0000 UTC m=+2251.482032489"
Jul 23 00:45:17.623789 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:17.623747    2565 generic.go:356] "Generic (PLEG): container finished" podID="feeb9da4-a330-4699-a1a4-c897972b3937" containerID="9c139a264654779197fa152ab654944f9c0e183673edc90593295e6dfe7fbbab" exitCode=0
Jul 23 00:45:17.624211 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:17.623827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz" event={"ID":"feeb9da4-a330-4699-a1a4-c897972b3937","Type":"ContainerDied","Data":"9c139a264654779197fa152ab654944f9c0e183673edc90593295e6dfe7fbbab"}
Jul 23 00:45:18.761656 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:18.761633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:18.930350 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:18.930272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca\") pod \"feeb9da4-a330-4699-a1a4-c897972b3937\" (UID: \"feeb9da4-a330-4699-a1a4-c897972b3937\") "
Jul 23 00:45:18.930649 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:18.930625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "feeb9da4-a330-4699-a1a4-c897972b3937" (UID: "feeb9da4-a330-4699-a1a4-c897972b3937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:45:19.031614 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:19.031587    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feeb9da4-a330-4699-a1a4-c897972b3937-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:45:19.634443 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:19.634413    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"
Jul 23 00:45:19.634626 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:19.634417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz" event={"ID":"feeb9da4-a330-4699-a1a4-c897972b3937","Type":"ContainerDied","Data":"d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7"}
Jul 23 00:45:19.634626 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:19.634532    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d55e94c770b54f93757d54bc524d8464d9a12a94a9833252bf2a642664156fc7"
Jul 23 00:45:29.210583 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:29.210551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:45:30.916615 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:30.916582    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"]
Jul 23 00:45:30.921569 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:30.921542    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c65690b4-p4twz"]
Jul 23 00:45:31.726051 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:45:31.726018    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feeb9da4-a330-4699-a1a4-c897972b3937" path="/var/lib/kubelet/pods/feeb9da4-a330-4699-a1a4-c897972b3937/volumes"
Jul 23 00:46:06.555526 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:06.555489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:46:19.027511 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:19.027480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:46:36.427808 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:36.427762    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:46:42.612933 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.612847    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"]
Jul 23 00:46:42.613421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.613262    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feeb9da4-a330-4699-a1a4-c897972b3937" containerName="revoke-keys"
Jul 23 00:46:42.613421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.613277    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="feeb9da4-a330-4699-a1a4-c897972b3937" containerName="revoke-keys"
Jul 23 00:46:42.613421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.613336    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="feeb9da4-a330-4699-a1a4-c897972b3937" containerName="revoke-keys"
Jul 23 00:46:42.616193 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.616176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:42.618616 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.618584    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:46:42.618758 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.618634    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:46:42.623001 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.622945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"]
Jul 23 00:46:42.639896 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.639866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76cb26bd-9f462\" (UID: \"0048deab-25f8-49a8-bc60-da8cf1ac63b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:42.740987 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.740934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76cb26bd-9f462\" (UID: \"0048deab-25f8-49a8-bc60-da8cf1ac63b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:42.741585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.741558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76cb26bd-9f462\" (UID: \"0048deab-25f8-49a8-bc60-da8cf1ac63b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:42.928107 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:42.928014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:43.066097 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:43.066006    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"]
Jul 23 00:46:43.069329 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:46:43.069295    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0048deab_25f8_49a8_bc60_da8cf1ac63b9.slice/crio-c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3 WatchSource:0}: Error finding container c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3: Status 404 returned error can't find the container with id c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3
Jul 23 00:46:44.049440 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:44.049402    2565 generic.go:356] "Generic (PLEG): container finished" podID="0048deab-25f8-49a8-bc60-da8cf1ac63b9" containerID="b0fc2adee99f743015b00746eda86ca24de2e3d5e1152c72060c983bad95e9bc" exitCode=0
Jul 23 00:46:44.049817 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:44.049483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462" event={"ID":"0048deab-25f8-49a8-bc60-da8cf1ac63b9","Type":"ContainerDied","Data":"b0fc2adee99f743015b00746eda86ca24de2e3d5e1152c72060c983bad95e9bc"}
Jul 23 00:46:44.049817 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:44.049537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462" event={"ID":"0048deab-25f8-49a8-bc60-da8cf1ac63b9","Type":"ContainerStarted","Data":"c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3"}
Jul 23 00:46:45.199292 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:45.199265    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:45.263398 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:45.263366    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca\") pod \"0048deab-25f8-49a8-bc60-da8cf1ac63b9\" (UID: \"0048deab-25f8-49a8-bc60-da8cf1ac63b9\") "
Jul 23 00:46:45.263772 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:45.263743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0048deab-25f8-49a8-bc60-da8cf1ac63b9" (UID: "0048deab-25f8-49a8-bc60-da8cf1ac63b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:46:45.364886 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:45.364803    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0048deab-25f8-49a8-bc60-da8cf1ac63b9-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:46:46.062264 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:46.062189    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"
Jul 23 00:46:46.062417 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:46.062187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462" event={"ID":"0048deab-25f8-49a8-bc60-da8cf1ac63b9","Type":"ContainerDied","Data":"c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3"}
Jul 23 00:46:46.062417 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:46.062313    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a3cf80547a394f7588af4a9fbb45eaf11ff6915854ded98fda365a587c02f3"
Jul 23 00:46:56.816866 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:46:56.816833    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:47:02.699342 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:47:02.699308    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"]
Jul 23 00:47:02.701053 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:47:02.701030    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76cb26bd-9f462"]
Jul 23 00:47:03.726036 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:47:03.726005    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0048deab-25f8-49a8-bc60-da8cf1ac63b9" path="/var/lib/kubelet/pods/0048deab-25f8-49a8-bc60-da8cf1ac63b9/volumes"
Jul 23 00:47:07.022696 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:47:07.022664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:47:47.747593 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:47:47.747562    2565 scope.go:117] "RemoveContainer" containerID="95edcef4ab9f1f0e64b74b6e82facff5432a7b6a2ea2e4c786c5aa214d93c34b"
Jul 23 00:48:07.529394 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:48:07.529356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:48:19.725538 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:48:19.725506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:48:47.766175 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:48:47.766139    2565 scope.go:117] "RemoveContainer" containerID="b2b9f7fd29c683da0ca5a67c2bd69d539e780eef3f57bdf3e31dd7e1c175104b"
Jul 23 00:49:07.335637 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:07.335599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:49:18.617145 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:18.617109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:49:27.841303 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.841273    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:49:27.841698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.841614    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0048deab-25f8-49a8-bc60-da8cf1ac63b9" containerName="revoke-keys"
Jul 23 00:49:27.841698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.841625    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0048deab-25f8-49a8-bc60-da8cf1ac63b9" containerName="revoke-keys"
Jul 23 00:49:27.841698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.841688    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0048deab-25f8-49a8-bc60-da8cf1ac63b9" containerName="revoke-keys"
Jul 23 00:49:27.844246 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.844222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:27.847137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.847110    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5bc11\"/\"default-dockercfg-mp757\""
Jul 23 00:49:27.848378 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.848357    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5bc11\"/\"kube-root-ca.crt\""
Jul 23 00:49:27.848586 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.848567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5bc11\"/\"test-model-e5bc11-kserve-self-signed-certs\""
Jul 23 00:49:27.848667 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.848611    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5bc11\"/\"openshift-service-ca.crt\""
Jul 23 00:49:27.855689 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:27.855666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:49:28.017036 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.016997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.017205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.017056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.017205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.017097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.017205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.017123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggs6\" (UniqueName: \"kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.017205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.017181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.017360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.017288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.118764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.118764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.118838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggs6\" (UniqueName: \"kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.119171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.119185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.119252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.119232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.121458 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.121434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.121458 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.121457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.130755 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.130731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggs6\" (UniqueName: \"kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6\") pod \"test-model-e5bc11-kserve-c5bf76bc9-xmxhq\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") " pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.154256 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.154235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:28.290655 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.290621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:49:28.291879 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:49:28.291852    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3494342_21cc_4e75_93ad_967e188594a0.slice/crio-9777b4126968a875e8a102bd326e435c4e53cca73b16c17e97950f5ec0ec0c17 WatchSource:0}: Error finding container 9777b4126968a875e8a102bd326e435c4e53cca73b16c17e97950f5ec0ec0c17: Status 404 returned error can't find the container with id 9777b4126968a875e8a102bd326e435c4e53cca73b16c17e97950f5ec0ec0c17
Jul 23 00:49:28.835446 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.835415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" event={"ID":"b3494342-21cc-4e75-93ad-967e188594a0","Type":"ContainerStarted","Data":"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"}
Jul 23 00:49:28.835619 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.835456    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" event={"ID":"b3494342-21cc-4e75-93ad-967e188594a0","Type":"ContainerStarted","Data":"9777b4126968a875e8a102bd326e435c4e53cca73b16c17e97950f5ec0ec0c17"}
Jul 23 00:49:28.855560 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:28.855512    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" podStartSLOduration=1.855497682 podStartE2EDuration="1.855497682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:28.853964892 +0000 UTC m=+2503.707556132" watchObservedRunningTime="2026-07-23 00:49:28.855497682 +0000 UTC m=+2503.709067354"
Jul 23 00:49:31.520934 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:31.520897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:49:38.154560 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:38.154480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:38.154560 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:38.154525    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:38.166698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:38.166673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:38.894135 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:38.894110    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:49:43.730252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:43.730222    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:49:46.730698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.730655    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:49:46.734857 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.734838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.737756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.737736    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a242b2\"/\"default-dockercfg-m7xxh\""
Jul 23 00:49:46.737756 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.737748    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a242b2\"/\"kube-root-ca.crt\""
Jul 23 00:49:46.737943 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.737736    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a242b2\"/\"openshift-service-ca.crt\""
Jul 23 00:49:46.738996 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.738954    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a242b2\"/\"test-model-a242b2-kserve-self-signed-certs\""
Jul 23 00:49:46.741994 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.741956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:49:46.760987 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.760941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njh9x\" (UniqueName: \"kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.761081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.761009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.761081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.761035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.761081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.761077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.761195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.761097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.761195 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.761135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862226 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862226 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862448 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njh9x\" (UniqueName: \"kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.862876 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.862791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.864446 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.864423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.864713 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.864696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:46.870730 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:46.870709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njh9x\" (UniqueName: \"kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x\") pod \"test-model-a242b2-kserve-6c9d8f49c9-rdl4z\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") " pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:47.044897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.044860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:47.175087 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.175064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:49:47.176842 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:49:47.176811    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaa722f4_dcc4_4490_af01_83248eb78626.slice/crio-2e0936b9eb9dc8b6dfa6b3c72c9b3c531a617abb7a9261102811e9869046066f WatchSource:0}: Error finding container 2e0936b9eb9dc8b6dfa6b3c72c9b3c531a617abb7a9261102811e9869046066f: Status 404 returned error can't find the container with id 2e0936b9eb9dc8b6dfa6b3c72c9b3c531a617abb7a9261102811e9869046066f
Jul 23 00:49:47.784134 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.784102    2565 scope.go:117] "RemoveContainer" containerID="557288dca0cd7901c2207f7649bb0698505368bd46c9d7fad481a3a7212b23f8"
Jul 23 00:49:47.926284 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.926244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" event={"ID":"daa722f4-dcc4-4490-af01-83248eb78626","Type":"ContainerStarted","Data":"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"}
Jul 23 00:49:47.926284 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.926287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" event={"ID":"daa722f4-dcc4-4490-af01-83248eb78626","Type":"ContainerStarted","Data":"2e0936b9eb9dc8b6dfa6b3c72c9b3c531a617abb7a9261102811e9869046066f"}
Jul 23 00:49:47.946459 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:47.946367    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" podStartSLOduration=1.946354369 podStartE2EDuration="1.946354369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:49:47.946139225 +0000 UTC m=+2522.799708910" watchObservedRunningTime="2026-07-23 00:49:47.946354369 +0000 UTC m=+2522.799924092"
Jul 23 00:49:54.183645 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:54.183610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:49:57.045688 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:57.045656    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:57.046128 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:57.045718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:57.058056 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:57.058031    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:49:57.987198 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:49:57.987169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:50:34.907512 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:34.907476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:50:35.815570 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:35.815537    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:50:36.019534 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.019502    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"]
Jul 23 00:50:36.022802 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.022785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:36.027121 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.027104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:50:36.031720 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.031701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:50:36.047592 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.047569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"]
Jul 23 00:50:36.094692 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.094614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h\" (UID: \"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:36.195701 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.195666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h\" (UID: \"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:36.196360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.196335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h\" (UID: \"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:36.333152 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.333128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:36.467764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:36.467587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"]
Jul 23 00:50:36.470298 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:50:36.470259    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee8fe05_dd03_4a75_85dc_e8bbb4d182e7.slice/crio-a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451 WatchSource:0}: Error finding container a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451: Status 404 returned error can't find the container with id a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451
Jul 23 00:50:37.160816 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:37.160774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h" event={"ID":"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7","Type":"ContainerStarted","Data":"91c9c1e527e3fa48d5c20f5aabdc33da035a070604c860c253b17df1c1b91263"}
Jul 23 00:50:37.161232 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:37.160836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h" event={"ID":"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7","Type":"ContainerStarted","Data":"a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451"}
Jul 23 00:50:37.177806 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:37.177758    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h" podStartSLOduration=2.177744171 podStartE2EDuration="2.177744171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:50:37.175539165 +0000 UTC m=+2572.029108849" watchObservedRunningTime="2026-07-23 00:50:37.177744171 +0000 UTC m=+2572.031313842"
Jul 23 00:50:38.168042 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:38.168007    2565 generic.go:356] "Generic (PLEG): container finished" podID="fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" containerID="91c9c1e527e3fa48d5c20f5aabdc33da035a070604c860c253b17df1c1b91263" exitCode=0
Jul 23 00:50:38.168417 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:38.168093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h" event={"ID":"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7","Type":"ContainerDied","Data":"91c9c1e527e3fa48d5c20f5aabdc33da035a070604c860c253b17df1c1b91263"}
Jul 23 00:50:39.304910 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:39.304882    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:39.322023 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:39.321993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca\") pod \"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7\" (UID: \"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7\") "
Jul 23 00:50:39.322528 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:39.322498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" (UID: "fee8fe05-dd03-4a75-85dc-e8bbb4d182e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:50:39.423034 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:39.423001    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:40.028610 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:40.028577    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:50:40.181788 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:40.181751    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"
Jul 23 00:50:40.181952 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:40.181793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h" event={"ID":"fee8fe05-dd03-4a75-85dc-e8bbb4d182e7","Type":"ContainerDied","Data":"a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451"}
Jul 23 00:50:40.181952 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:40.181858    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a36aa47937d4aeb1f56759f7f1343ab5fdaa381c17c37087e323950e4469f451"
Jul 23 00:50:49.927581 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:49.927534    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" podUID="b3494342-21cc-4e75-93ad-967e188594a0" containerName="main" containerID="cri-o://26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca" gracePeriod=45
Jul 23 00:50:50.179845 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.179783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:50:50.214746 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.214898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214781    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.214898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bggs6\" (UniqueName: \"kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.214898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.215062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.215062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.214960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm\") pod \"b3494342-21cc-4e75-93ad-967e188594a0\" (UID: \"b3494342-21cc-4e75-93ad-967e188594a0\") "
Jul 23 00:50:50.215062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.215025    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:50.215062 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.215048    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:50.215337 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.215233    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home" (OuterVolumeSpecName: "home") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:50.215337 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.215295    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.215337 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.215316    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.217210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.217184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:50:50.217210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.217198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm" (OuterVolumeSpecName: "dshm") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:50.217210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.217199    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6" (OuterVolumeSpecName: "kube-api-access-bggs6") pod "b3494342-21cc-4e75-93ad-967e188594a0" (UID: "b3494342-21cc-4e75-93ad-967e188594a0"). InnerVolumeSpecName "kube-api-access-bggs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:50:50.231194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.231171    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3494342-21cc-4e75-93ad-967e188594a0" containerID="26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca" exitCode=0
Jul 23 00:50:50.231292 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.231212    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"
Jul 23 00:50:50.231292 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.231259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" event={"ID":"b3494342-21cc-4e75-93ad-967e188594a0","Type":"ContainerDied","Data":"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"}
Jul 23 00:50:50.231376 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.231298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq" event={"ID":"b3494342-21cc-4e75-93ad-967e188594a0","Type":"ContainerDied","Data":"9777b4126968a875e8a102bd326e435c4e53cca73b16c17e97950f5ec0ec0c17"}
Jul 23 00:50:50.231376 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.231315    2565 scope.go:117] "RemoveContainer" containerID="26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"
Jul 23 00:50:50.239567 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.239552    2565 scope.go:117] "RemoveContainer" containerID="26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"
Jul 23 00:50:50.239815 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:50:50.239798    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca\": container with ID starting with 26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca not found: ID does not exist" containerID="26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"
Jul 23 00:50:50.239861 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.239824    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca"} err="failed to get container status \"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca\": rpc error: code = NotFound desc = could not find container \"26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca\": container with ID starting with 26174de2c4a9327b6aff53b52ac7c79c5a7f82094701dad7fe3f3f27760fc9ca not found: ID does not exist"
Jul 23 00:50:50.253388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.253361    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:50:50.257282 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.257262    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e5bc11/test-model-e5bc11-kserve-c5bf76bc9-xmxhq"]
Jul 23 00:50:50.316648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.316616    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bggs6\" (UniqueName: \"kubernetes.io/projected/b3494342-21cc-4e75-93ad-967e188594a0-kube-api-access-bggs6\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.316648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.316644    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3494342-21cc-4e75-93ad-967e188594a0-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.316799 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.316662    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.316799 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.316673    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3494342-21cc-4e75-93ad-967e188594a0-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:50.836103 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:50.836060    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" podUID="daa722f4-dcc4-4490-af01-83248eb78626" containerName="main" containerID="cri-o://c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859" gracePeriod=45
Jul 23 00:50:51.098359 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.098271    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"]
Jul 23 00:50:51.100354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.100330    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e5bc11-fzw2h"]
Jul 23 00:50:51.103076 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.103056    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:50:51.224525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224583    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njh9x\" (UniqueName: \"kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224758    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm\") pod \"daa722f4-dcc4-4490-af01-83248eb78626\" (UID: \"daa722f4-dcc4-4490-af01-83248eb78626\") "
Jul 23 00:50:51.224889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.224791    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home" (OuterVolumeSpecName: "home") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:51.225070 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.225011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache" (OuterVolumeSpecName: "model-cache") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:51.225070 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.225023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:51.225148 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.225078    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.227015 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.226934    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm" (OuterVolumeSpecName: "dshm") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:50:51.227015 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.227008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:50:51.227163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.227027    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x" (OuterVolumeSpecName: "kube-api-access-njh9x") pod "daa722f4-dcc4-4490-af01-83248eb78626" (UID: "daa722f4-dcc4-4490-af01-83248eb78626"). InnerVolumeSpecName "kube-api-access-njh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:50:51.238476 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.238451    2565 generic.go:356] "Generic (PLEG): container finished" podID="daa722f4-dcc4-4490-af01-83248eb78626" containerID="c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859" exitCode=0
Jul 23 00:50:51.238568 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.238492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" event={"ID":"daa722f4-dcc4-4490-af01-83248eb78626","Type":"ContainerDied","Data":"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"}
Jul 23 00:50:51.238568 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.238509    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"
Jul 23 00:50:51.238568 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.238528    2565 scope.go:117] "RemoveContainer" containerID="c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"
Jul 23 00:50:51.238693 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.238518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z" event={"ID":"daa722f4-dcc4-4490-af01-83248eb78626","Type":"ContainerDied","Data":"2e0936b9eb9dc8b6dfa6b3c72c9b3c531a617abb7a9261102811e9869046066f"}
Jul 23 00:50:51.247013 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.246990    2565 scope.go:117] "RemoveContainer" containerID="c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"
Jul 23 00:50:51.247261 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:50:51.247243    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859\": container with ID starting with c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859 not found: ID does not exist" containerID="c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"
Jul 23 00:50:51.247334 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.247270    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859"} err="failed to get container status \"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859\": rpc error: code = NotFound desc = could not find container \"c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859\": container with ID starting with c16f683eac57383f33d1d8a1acf2512f918b72a7e65e0f36930d761e408cb859 not found: ID does not exist"
Jul 23 00:50:51.263950 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.263923    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:50:51.268203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.268168    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a242b2/test-model-a242b2-kserve-6c9d8f49c9-rdl4z"]
Jul 23 00:50:51.326251 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.326217    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa722f4-dcc4-4490-af01-83248eb78626-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.326251 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.326247    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njh9x\" (UniqueName: \"kubernetes.io/projected/daa722f4-dcc4-4490-af01-83248eb78626-kube-api-access-njh9x\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.326251 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.326259    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.326501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.326270    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.326501 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.326279    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa722f4-dcc4-4490-af01-83248eb78626-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:50:51.726072 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.726040    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3494342-21cc-4e75-93ad-967e188594a0" path="/var/lib/kubelet/pods/b3494342-21cc-4e75-93ad-967e188594a0/volumes"
Jul 23 00:50:51.726443 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.726429    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa722f4-dcc4-4490-af01-83248eb78626" path="/var/lib/kubelet/pods/daa722f4-dcc4-4490-af01-83248eb78626/volumes"
Jul 23 00:50:51.726712 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:51.726701    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" path="/var/lib/kubelet/pods/fee8fe05-dd03-4a75-85dc-e8bbb4d182e7/volumes"
Jul 23 00:50:54.984518 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:54.984480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:50:58.178270 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178233    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"]
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178605    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3494342-21cc-4e75-93ad-967e188594a0" containerName="main"
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178619    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3494342-21cc-4e75-93ad-967e188594a0" containerName="main"
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178647    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" containerName="revoke-keys"
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178657    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" containerName="revoke-keys"
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178666    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa722f4-dcc4-4490-af01-83248eb78626" containerName="main"
Jul 23 00:50:58.178678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178671    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa722f4-dcc4-4490-af01-83248eb78626" containerName="main"
Jul 23 00:50:58.178893 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178728    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3494342-21cc-4e75-93ad-967e188594a0" containerName="main"
Jul 23 00:50:58.178893 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178734    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa722f4-dcc4-4490-af01-83248eb78626" containerName="main"
Jul 23 00:50:58.178893 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.178740    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee8fe05-dd03-4a75-85dc-e8bbb4d182e7" containerName="revoke-keys"
Jul 23 00:50:58.181812 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.181795    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:50:58.184635 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.184612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:50:58.184770 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.184691    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:50:58.189776 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.189482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"]
Jul 23 00:50:58.285964 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.285931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6\" (UID: \"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:50:58.386498 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.386461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6\" (UID: \"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:50:58.387163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.387140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6\" (UID: \"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:50:58.494530 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.494484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:50:58.625507 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:58.625434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"]
Jul 23 00:50:58.628098 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:50:58.628068    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3848a7a_b01f_4e29_b8c5_ca9b8c4ce092.slice/crio-e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0 WatchSource:0}: Error finding container e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0: Status 404 returned error can't find the container with id e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0
Jul 23 00:50:59.287502 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:59.287468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6" event={"ID":"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092","Type":"ContainerStarted","Data":"78f6f385543521faa864c5c6233f1cc205f7da5dc60bf9fb39330d3bbe1b5295"}
Jul 23 00:50:59.287944 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:59.287525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6" event={"ID":"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092","Type":"ContainerStarted","Data":"e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0"}
Jul 23 00:50:59.324791 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:50:59.324725    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6" podStartSLOduration=1.324710462 podStartE2EDuration="1.324710462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:50:59.323320686 +0000 UTC m=+2594.176890358" watchObservedRunningTime="2026-07-23 00:50:59.324710462 +0000 UTC m=+2594.178280134"
Jul 23 00:51:00.294322 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:00.294288    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" containerID="78f6f385543521faa864c5c6233f1cc205f7da5dc60bf9fb39330d3bbe1b5295" exitCode=0
Jul 23 00:51:00.294731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:00.294345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6" event={"ID":"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092","Type":"ContainerDied","Data":"78f6f385543521faa864c5c6233f1cc205f7da5dc60bf9fb39330d3bbe1b5295"}
Jul 23 00:51:01.429558 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:01.429530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:51:01.514936 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:01.514890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca\") pod \"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092\" (UID: \"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092\") "
Jul 23 00:51:01.515272 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:01.515248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" (UID: "b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:51:01.615590 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:01.615501    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:51:02.304823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:02.304782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6" event={"ID":"b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092","Type":"ContainerDied","Data":"e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0"}
Jul 23 00:51:02.304823 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:02.304809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"
Jul 23 00:51:02.305117 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:02.304877    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f5271314bfc899dec3308cd76eefd552d663f538649097313fb9c6167f68c0"
Jul 23 00:51:11.529989 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:11.529889    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:51:13.257182 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:13.257147    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"]
Jul 23 00:51:13.259885 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:13.259845    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a242b2-ctff6"]
Jul 23 00:51:13.729741 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:13.729648    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" path="/var/lib/kubelet/pods/b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092/volumes"
Jul 23 00:51:20.684569 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.684531    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:51:20.684961 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.684913    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" containerName="revoke-keys"
Jul 23 00:51:20.684961 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.684925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" containerName="revoke-keys"
Jul 23 00:51:20.685083 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.685015    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3848a7a-b01f-4e29-b8c5-ca9b8c4ce092" containerName="revoke-keys"
Jul 23 00:51:20.689000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.688959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.692591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.692572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-16466f-openshift-default-dockercfg-tb58x\""
Jul 23 00:51:20.701583 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.701558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:51:20.783380 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783380 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64qd\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783608 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.783757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.783704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884727 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64qd\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.884888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885233 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885233 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.884936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885233 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.885175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885397 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.885234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885550 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.885528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.885735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.885866 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.885850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.887460 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.887445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.887697 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.887677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.893571 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.893550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.893708 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.893676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64qd\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd\") pod \"e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") " pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:20.999107 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:20.999084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:21.135469 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.135447    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:51:21.137365 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:51:21.137329    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9774980f_de35_4909_886d_17d7b5c5a1b4.slice/crio-4c6ab8d49669e82c389f8451927a6f353339a4c6d6145449d15b682b1fff3465 WatchSource:0}: Error finding container 4c6ab8d49669e82c389f8451927a6f353339a4c6d6145449d15b682b1fff3465: Status 404 returned error can't find the container with id 4c6ab8d49669e82c389f8451927a6f353339a4c6d6145449d15b682b1fff3465
Jul 23 00:51:21.139520 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.139487    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:21.139615 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.139551    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:21.139615 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.139580    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:21.403591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.403507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerStarted","Data":"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"}
Jul 23 00:51:21.403591 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.403562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerStarted","Data":"4c6ab8d49669e82c389f8451927a6f353339a4c6d6145449d15b682b1fff3465"}
Jul 23 00:51:21.423285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.423233    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podStartSLOduration=1.423217979 podStartE2EDuration="1.423217979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:51:21.422509368 +0000 UTC m=+2616.276079053" watchObservedRunningTime="2026-07-23 00:51:21.423217979 +0000 UTC m=+2616.276787652"
Jul 23 00:51:21.999321 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:21.999276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:23.005262 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:23.005233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:23.414631 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:23.414546    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:23.415502 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:23.415479    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:31.140380 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:31.140347    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:51:38.916095 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:38.916058    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:56494->10.132.0.55:15021: read: connection reset by peer" start-of-body=
Jul 23 00:51:38.916511 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:38.916151    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:56494->10.132.0.55:15021: read: connection reset by peer"
Jul 23 00:51:39.503452 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.503417    2565 generic.go:356] "Generic (PLEG): container finished" podID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerID="003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9" exitCode=0
Jul 23 00:51:39.503634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.503495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerDied","Data":"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"}
Jul 23 00:51:39.503962 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.503941    2565 scope.go:117] "RemoveContainer" containerID="003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"
Jul 23 00:51:39.504939 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.504910    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:39.505031 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.505017    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:39.505074 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:39.505054    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:51:40.511230 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:40.511199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerStarted","Data":"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"}
Jul 23 00:51:41.000252 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:41.000218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:41.000449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:41.000416    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 23 00:51:41.000495 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:41.000459    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 23 00:51:42.000095 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:42.000053    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 23 00:51:42.000646 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:42.000123    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 23 00:51:42.999764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:42.999720    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 23 00:51:42.999952 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:42.999786    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 23 00:51:43.999455 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:43.999418    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 23 00:51:43.999838 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:43.999475    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 23 00:51:45.003544 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:45.003518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:45.004032 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:45.004010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:45.004673 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:45.004650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:51:46.637383 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:46.637352    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:51:47.809452 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:51:47.809420    2565 scope.go:117] "RemoveContainer" containerID="9c139a264654779197fa152ab654944f9c0e183673edc90593295e6dfe7fbbab"
Jul 23 00:52:01.937509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:01.937462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:52:10.967182 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:10.967148    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:52:10.968047 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:10.968013    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" containerID="cri-o://bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb" gracePeriod=30
Jul 23 00:52:15.006347 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:15.006309    2565 patch_prober.go:28] interesting pod/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 00:52:15.006818 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:15.006377    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 00:52:16.217379 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.217358    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:52:16.291913 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.291879    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.291913 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.291918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.291937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.291963    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j64qd\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd\") pod \"9774980f-de35-4909-886d-17d7b5c5a1b4\" (UID: \"9774980f-de35-4909-886d-17d7b5c5a1b4\") "
Jul 23 00:52:16.292449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:52:16.292449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292364    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:52:16.292449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:52:16.292691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292543    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.292691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292564    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-workload-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.292691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292578    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-credential-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.292691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.292631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:52:16.293102 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.293076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:52:16.294355 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.294335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:52:16.294439 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.294421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd" (OuterVolumeSpecName: "kube-api-access-j64qd") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "kube-api-access-j64qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:52:16.294733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.294705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 00:52:16.294733 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.294722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9774980f-de35-4909-886d-17d7b5c5a1b4" (UID: "9774980f-de35-4909-886d-17d7b5c5a1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:52:16.393749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393708    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-podinfo\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.393749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393739    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64qd\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-kube-api-access-j64qd\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.393749 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393753    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-token\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.394016 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393766    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-envoy\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.394016 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393778    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9774980f-de35-4909-886d-17d7b5c5a1b4-istiod-ca-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.394016 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.393789    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9774980f-de35-4909-886d-17d7b5c5a1b4-istio-data\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:52:16.684437 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.684351    2565 generic.go:356] "Generic (PLEG): container finished" podID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerID="bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb" exitCode=0
Jul 23 00:52:16.684437 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.684400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"
Jul 23 00:52:16.684437 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.684418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerDied","Data":"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"}
Jul 23 00:52:16.684675 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.684464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2" event={"ID":"9774980f-de35-4909-886d-17d7b5c5a1b4","Type":"ContainerDied","Data":"4c6ab8d49669e82c389f8451927a6f353339a4c6d6145449d15b682b1fff3465"}
Jul 23 00:52:16.684675 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.684491    2565 scope.go:117] "RemoveContainer" containerID="bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"
Jul 23 00:52:16.693948 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.693926    2565 scope.go:117] "RemoveContainer" containerID="003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"
Jul 23 00:52:16.701853 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.701835    2565 scope.go:117] "RemoveContainer" containerID="bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"
Jul 23 00:52:16.702163 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:52:16.702143    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb\": container with ID starting with bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb not found: ID does not exist" containerID="bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"
Jul 23 00:52:16.702233 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.702173    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb"} err="failed to get container status \"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb\": rpc error: code = NotFound desc = could not find container \"bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb\": container with ID starting with bf508b850b087009f0d177992c06daeedcf09762d8094dff368e1f1be75153eb not found: ID does not exist"
Jul 23 00:52:16.702233 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.702197    2565 scope.go:117] "RemoveContainer" containerID="003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"
Jul 23 00:52:16.702486 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:52:16.702467    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9\": container with ID starting with 003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9 not found: ID does not exist" containerID="003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"
Jul 23 00:52:16.702563 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.702490    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9"} err="failed to get container status \"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9\": rpc error: code = NotFound desc = could not find container \"003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9\": container with ID starting with 003cbfb9c216b5f0d5e5b466c6d72d6b8ef33e8aa9262c3df352a29f5a38d2c9 not found: ID does not exist"
Jul 23 00:52:16.724242 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.724223    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:52:16.743300 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:16.743273    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-16466f-openshift-default-8fc5f6bdf-gfwp2"]
Jul 23 00:52:17.726113 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:17.726077    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" path="/var/lib/kubelet/pods/9774980f-de35-4909-886d-17d7b5c5a1b4/volumes"
Jul 23 00:52:32.030043 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:32.030009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.912709    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913594    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913623    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913654    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913671    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913887    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.915685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.913899    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774980f-de35-4909-886d-17d7b5c5a1b4" containerName="istio-proxy"
Jul 23 00:52:38.919631 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.919609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:38.922360 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.922335    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c75140-openshift-default-dockercfg-77cnx\""
Jul 23 00:52:38.932915 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.932891    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:52:38.999984 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.999931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:38.999989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000372 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000372 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.000372 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.000300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.101731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.101731 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102027 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.101905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102396 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.102230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102396 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.102370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102552 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.102526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102603 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.102566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.102721 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.102705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.104162 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.104133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.104379 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.104362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.110579 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.110551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.110707 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.110686    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt\") pod \"e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") " pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.229795 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.229713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:39.369479 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.369447    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:52:39.371693 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:52:39.371659    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc122d01d_018f_4dfe_b888_e2e693969cb1.slice/crio-58284acd61949a74610eb485399b2a5d4bf7b6be87b16c74cb441600cd9fa62b WatchSource:0}: Error finding container 58284acd61949a74610eb485399b2a5d4bf7b6be87b16c74cb441600cd9fa62b: Status 404 returned error can't find the container with id 58284acd61949a74610eb485399b2a5d4bf7b6be87b16c74cb441600cd9fa62b
Jul 23 00:52:39.373893 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.373849    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:39.374014 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.373944    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:39.374073 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.374054    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:39.798623 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.798587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerStarted","Data":"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"}
Jul 23 00:52:39.798789 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.798641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerStarted","Data":"58284acd61949a74610eb485399b2a5d4bf7b6be87b16c74cb441600cd9fa62b"}
Jul 23 00:52:39.821499 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:39.821446    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podStartSLOduration=1.821431759 podStartE2EDuration="1.821431759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:52:39.818763047 +0000 UTC m=+2694.672332720" watchObservedRunningTime="2026-07-23 00:52:39.821431759 +0000 UTC m=+2694.675001473"
Jul 23 00:52:40.230469 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:40.230381    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:40.235598 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:40.235573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:40.804691 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:40.804660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:40.805557 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:40.805537    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:46.955579 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:46.955548    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:52:47.835194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:47.835154    2565 scope.go:117] "RemoveContainer" containerID="b0fc2adee99f743015b00746eda86ca24de2e3d5e1152c72060c983bad95e9bc"
Jul 23 00:52:55.805897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.805858    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 23 00:52:55.806331 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.805943    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 23 00:52:55.877711 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.877672    2565 generic.go:356] "Generic (PLEG): container finished" podID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerID="92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9" exitCode=0
Jul 23 00:52:55.877863 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.877748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerDied","Data":"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"}
Jul 23 00:52:55.878143 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.878129    2565 scope.go:117] "RemoveContainer" containerID="92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"
Jul 23 00:52:55.879226 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.879194    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:55.879307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.879264    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:55.879307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:55.879300    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 23 00:52:56.886897 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:56.886850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerStarted","Data":"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"}
Jul 23 00:52:57.230285 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:57.230196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:52:57.230449 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:57.230415    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 23 00:52:57.230494 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:57.230471    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 23 00:52:58.230275 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:58.230243    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 23 00:52:58.230649 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:58.230299    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 23 00:52:59.231010 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:59.230956    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 23 00:52:59.231376 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:52:59.231045    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 23 00:53:00.230293 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:00.230258    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 23 00:53:00.230478 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:00.230317    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 23 00:53:01.234362 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:01.234332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:53:01.234767 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:01.234566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:53:01.235238 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:01.235216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:53:02.633416 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:02.633382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:53:04.810134 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.810101    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:53:04.813219 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.813194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.815865 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.815834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"openshift-service-ca.crt\""
Jul 23 00:53:04.816937 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.816916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"default-dockercfg-xcq7r\""
Jul 23 00:53:04.817081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.816922    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"kube-root-ca.crt\""
Jul 23 00:53:04.817163 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.817123    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"auth-test-model-c75140-kserve-self-signed-certs\""
Jul 23 00:53:04.823253 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.823233    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:53:04.840277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.840423 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.840423 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.840548 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.840548 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnr5q\" (UniqueName: \"kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.840548 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.840516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941695 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.941899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.941878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnr5q\" (UniqueName: \"kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.942286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.942264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.942286 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.942281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.942380 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.942254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.944044 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.944021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.944307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.944291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:04.950241 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:04.950216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnr5q\" (UniqueName: \"kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q\") pod \"auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") " pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:05.123936 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:05.123861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:05.262584 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:05.262561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:53:05.264243 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:53:05.264211    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod272da39a_5073_411e_88e3_8f62ec8db80d.slice/crio-98ec044a11d68d295a04171fc7a37b3b86b1d88ebb831d9c1164b7aaa4834c13 WatchSource:0}: Error finding container 98ec044a11d68d295a04171fc7a37b3b86b1d88ebb831d9c1164b7aaa4834c13: Status 404 returned error can't find the container with id 98ec044a11d68d295a04171fc7a37b3b86b1d88ebb831d9c1164b7aaa4834c13
Jul 23 00:53:05.937014 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:05.936961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" event={"ID":"272da39a-5073-411e-88e3-8f62ec8db80d","Type":"ContainerStarted","Data":"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"}
Jul 23 00:53:05.937396 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:05.937025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" event={"ID":"272da39a-5073-411e-88e3-8f62ec8db80d","Type":"ContainerStarted","Data":"98ec044a11d68d295a04171fc7a37b3b86b1d88ebb831d9c1164b7aaa4834c13"}
Jul 23 00:53:05.956429 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:05.956384    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" podStartSLOduration=1.956370013 podStartE2EDuration="1.956370013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:53:05.954936804 +0000 UTC m=+2720.808506478" watchObservedRunningTime="2026-07-23 00:53:05.956370013 +0000 UTC m=+2720.809939686"
Jul 23 00:53:10.035485 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:10.035450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:53:15.124245 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:15.124206    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:15.124639 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:15.124265    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:15.136457 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:15.136431    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:15.996517 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:15.996488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:53:48.713211 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.713176    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:53:48.716779 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.716762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.720951 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.720923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"kube-root-ca.crt\""
Jul 23 00:53:48.721203 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.720934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"auth-test-model-d29ed6-kserve-self-signed-certs\""
Jul 23 00:53:48.721348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.720993    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"default-dockercfg-fvvtc\""
Jul 23 00:53:48.721348 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.720997    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"openshift-service-ca.crt\""
Jul 23 00:53:48.727770 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.727743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:53:48.828127 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.828127 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.828385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.828385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.828385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.828385 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.828381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9vn\" (UniqueName: \"kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9vn\" (UniqueName: \"kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929525 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929683 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929683 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929816 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.929941 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.929907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.931693 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.931669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.932081 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.932057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:48.940175 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:48.940153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9vn\" (UniqueName: \"kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn\") pod \"auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") " pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:49.027485 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:49.027461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:49.162084 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:49.162054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:53:49.163679 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:53:49.163650    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad34b9e6_ae1f_48a7_8252_33423c2efbcf.slice/crio-752b306231ccd9ba6ef416b052ed7dd259cecbcb480857994333938f54b2782b WatchSource:0}: Error finding container 752b306231ccd9ba6ef416b052ed7dd259cecbcb480857994333938f54b2782b: Status 404 returned error can't find the container with id 752b306231ccd9ba6ef416b052ed7dd259cecbcb480857994333938f54b2782b
Jul 23 00:53:50.147916 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:50.147876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" event={"ID":"ad34b9e6-ae1f-48a7-8252-33423c2efbcf","Type":"ContainerStarted","Data":"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"}
Jul 23 00:53:50.148337 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:50.147929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" event={"ID":"ad34b9e6-ae1f-48a7-8252-33423c2efbcf","Type":"ContainerStarted","Data":"752b306231ccd9ba6ef416b052ed7dd259cecbcb480857994333938f54b2782b"}
Jul 23 00:53:50.167411 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:50.167364    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" podStartSLOduration=2.167349743 podStartE2EDuration="2.167349743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:53:50.166420315 +0000 UTC m=+2765.019989989" watchObservedRunningTime="2026-07-23 00:53:50.167349743 +0000 UTC m=+2765.020919461"
Jul 23 00:53:52.531712 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:52.531675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:53:59.027664 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:59.027638    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:59.028175 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:59.027681    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:59.039648 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:59.039625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:53:59.201858 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:53:59.201832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:54:09.336606 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:54:09.336524    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:54:26.932597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:54:26.932559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:54:43.946755 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:54:43.946722    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:55:01.125450 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:01.125407    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:55:20.146364 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:20.146324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:55:39.140774 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:39.140680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:55:50.390913 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:50.390880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:55:51.240201 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:51.240172    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:55:55.931578 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:55:55.931529    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:56:05.410596 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.410544    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" podUID="272da39a-5073-411e-88e3-8f62ec8db80d" containerName="main" containerID="cri-o://7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43" gracePeriod=45
Jul 23 00:56:05.657210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.657187    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:56:05.666055 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.665996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666055 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home" (OuterVolumeSpecName: "home") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:05.666221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnr5q\" (UniqueName: \"kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666416 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache\") pod \"272da39a-5073-411e-88e3-8f62ec8db80d\" (UID: \"272da39a-5073-411e-88e3-8f62ec8db80d\") "
Jul 23 00:56:05.666416 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666373    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:05.666537 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666522    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.666597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666567    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.666597 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.666571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache" (OuterVolumeSpecName: "model-cache") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:05.668398 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.668370    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q" (OuterVolumeSpecName: "kube-api-access-jnr5q") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "kube-api-access-jnr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:56:05.668699 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.668665    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:56:05.668699 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.668675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm" (OuterVolumeSpecName: "dshm") pod "272da39a-5073-411e-88e3-8f62ec8db80d" (UID: "272da39a-5073-411e-88e3-8f62ec8db80d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:05.767512 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.767474    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnr5q\" (UniqueName: \"kubernetes.io/projected/272da39a-5073-411e-88e3-8f62ec8db80d-kube-api-access-jnr5q\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.767512 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.767506    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.767512 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.767515    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272da39a-5073-411e-88e3-8f62ec8db80d-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.767512 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.767523    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272da39a-5073-411e-88e3-8f62ec8db80d-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:05.812847 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.812810    2565 generic.go:356] "Generic (PLEG): container finished" podID="272da39a-5073-411e-88e3-8f62ec8db80d" containerID="7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43" exitCode=0
Jul 23 00:56:05.813030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.812861    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"
Jul 23 00:56:05.813030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.812895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" event={"ID":"272da39a-5073-411e-88e3-8f62ec8db80d","Type":"ContainerDied","Data":"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"}
Jul 23 00:56:05.813030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.812932    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8" event={"ID":"272da39a-5073-411e-88e3-8f62ec8db80d","Type":"ContainerDied","Data":"98ec044a11d68d295a04171fc7a37b3b86b1d88ebb831d9c1164b7aaa4834c13"}
Jul 23 00:56:05.813030 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.812956    2565 scope.go:117] "RemoveContainer" containerID="7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"
Jul 23 00:56:05.821428 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.821405    2565 scope.go:117] "RemoveContainer" containerID="7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"
Jul 23 00:56:05.821694 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:56:05.821672    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43\": container with ID starting with 7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43 not found: ID does not exist" containerID="7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"
Jul 23 00:56:05.821779 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.821718    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43"} err="failed to get container status \"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43\": rpc error: code = NotFound desc = could not find container \"7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43\": container with ID starting with 7357eabb0b534afd9f97a717c751b338863c0afef43a443f907eff70643dba43 not found: ID does not exist"
Jul 23 00:56:05.832330 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.832301    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:56:05.836561 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:05.836535    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/auth-test-model-c75140-kserve-6b69bdcdcf-wxdd8"]
Jul 23 00:56:06.259862 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.259822    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" podUID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" containerName="main" containerID="cri-o://754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2" gracePeriod=45
Jul 23 00:56:06.448642 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.448613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:56:06.514258 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.514199    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:56:06.574544 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.574728 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.574796 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.574796 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr9vn\" (UniqueName: \"kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.574901 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574794    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.574901 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.574817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache\") pod \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\" (UID: \"ad34b9e6-ae1f-48a7-8252-33423c2efbcf\") "
Jul 23 00:56:06.575044 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.575019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home" (OuterVolumeSpecName: "home") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:06.575149 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.575132    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.575249 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.575229    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:06.575309 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.575252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:06.576898 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.576875    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:56:06.577092 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.577073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm" (OuterVolumeSpecName: "dshm") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:56:06.577165 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.577130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn" (OuterVolumeSpecName: "kube-api-access-fr9vn") pod "ad34b9e6-ae1f-48a7-8252-33423c2efbcf" (UID: "ad34b9e6-ae1f-48a7-8252-33423c2efbcf"). InnerVolumeSpecName "kube-api-access-fr9vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:56:06.676516 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.676478    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr9vn\" (UniqueName: \"kubernetes.io/projected/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-kube-api-access-fr9vn\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.676516 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.676515    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.676516 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.676531    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.676772 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.676541    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.676772 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.676552    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34b9e6-ae1f-48a7-8252-33423c2efbcf-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:56:06.818672 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.818586    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" containerID="754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2" exitCode=0
Jul 23 00:56:06.818672 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.818637    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"
Jul 23 00:56:06.818890 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.818671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" event={"ID":"ad34b9e6-ae1f-48a7-8252-33423c2efbcf","Type":"ContainerDied","Data":"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"}
Jul 23 00:56:06.818890 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.818716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z" event={"ID":"ad34b9e6-ae1f-48a7-8252-33423c2efbcf","Type":"ContainerDied","Data":"752b306231ccd9ba6ef416b052ed7dd259cecbcb480857994333938f54b2782b"}
Jul 23 00:56:06.818890 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.818739    2565 scope.go:117] "RemoveContainer" containerID="754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"
Jul 23 00:56:06.827865 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.827846    2565 scope.go:117] "RemoveContainer" containerID="754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"
Jul 23 00:56:06.828152 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:56:06.828131    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2\": container with ID starting with 754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2 not found: ID does not exist" containerID="754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"
Jul 23 00:56:06.828208 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.828161    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2"} err="failed to get container status \"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2\": rpc error: code = NotFound desc = could not find container \"754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2\": container with ID starting with 754c1f462d1174e672ddbd5fd1e732dd8ef14d1d2a778d070463c1a8c5b602a2 not found: ID does not exist"
Jul 23 00:56:06.841677 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.841649    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:56:06.844685 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:06.844664    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/auth-test-model-d29ed6-kserve-5d7cc5c95f-t8t9z"]
Jul 23 00:56:07.725789 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:07.725757    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272da39a-5073-411e-88e3-8f62ec8db80d" path="/var/lib/kubelet/pods/272da39a-5073-411e-88e3-8f62ec8db80d/volumes"
Jul 23 00:56:07.726214 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:07.726165    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" path="/var/lib/kubelet/pods/ad34b9e6-ae1f-48a7-8252-33423c2efbcf/volumes"
Jul 23 00:56:20.540218 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540181    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:56:20.540619 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540553    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272da39a-5073-411e-88e3-8f62ec8db80d" containerName="main"
Jul 23 00:56:20.540619 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="272da39a-5073-411e-88e3-8f62ec8db80d" containerName="main"
Jul 23 00:56:20.540619 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" containerName="main"
Jul 23 00:56:20.540619 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540579    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" containerName="main"
Jul 23 00:56:20.540757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540633    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="272da39a-5073-411e-88e3-8f62ec8db80d" containerName="main"
Jul 23 00:56:20.540757 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.540644    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad34b9e6-ae1f-48a7-8252-33423c2efbcf" containerName="main"
Jul 23 00:56:20.543896 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.543880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.546931 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.546905    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"openshift-service-ca.crt\""
Jul 23 00:56:20.547096 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.546948    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"kube-root-ca.crt\""
Jul 23 00:56:20.548889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.548865    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"sub-test-model-d29ed6-kserve-self-signed-certs\""
Jul 23 00:56:20.548889 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.548880    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d29ed6\"/\"default-dockercfg-fvvtc\""
Jul 23 00:56:20.554000 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.553962    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:56:20.589388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.589513 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.589513 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.589607 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64sc\" (UniqueName: \"kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.589607 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.589688 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.589612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690253 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690253 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690483 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690483 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690483 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690483 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64sc\" (UniqueName: \"kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690840 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690935 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.690935 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.690907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.692489 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.692472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.692877 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.692856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.702846 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.702820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64sc\" (UniqueName: \"kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc\") pod \"sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") " pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.853987 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.853892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:20.984610 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:20.984583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:56:21.893539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:21.893504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" event={"ID":"e6763842-91f3-4610-a7a3-9e18c819bcd1","Type":"ContainerStarted","Data":"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"}
Jul 23 00:56:21.893539 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:21.893545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" event={"ID":"e6763842-91f3-4610-a7a3-9e18c819bcd1","Type":"ContainerStarted","Data":"26299416885b5046bab31bdce61051c515395215588f511ae09b2597d33cbd99"}
Jul 23 00:56:21.911954 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:21.911897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" podStartSLOduration=1.911879873 podStartE2EDuration="1.911879873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:56:21.911779514 +0000 UTC m=+2916.765349190" watchObservedRunningTime="2026-07-23 00:56:21.911879873 +0000 UTC m=+2916.765449547"
Jul 23 00:56:24.722644 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:24.722608    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:56:30.854445 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:30.854409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:30.854909 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:30.854466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:30.867060 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:30.867032    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:30.947620 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:30.947592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:56:38.936569 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:38.936528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:56:47.879366 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:47.879332    2565 scope.go:117] "RemoveContainer" containerID="91c9c1e527e3fa48d5c20f5aabdc33da035a070604c860c253b17df1c1b91263"
Jul 23 00:56:59.129764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:56:59.129728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:57:06.368625 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:06.365432    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:57:15.335593 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.335515    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:57:15.339183 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.339162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.342047 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.342025    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"kube-root-ca.crt\""
Jul 23 00:57:15.342186 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.342162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"default-dockercfg-xcq7r\""
Jul 23 00:57:15.342272 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.342254    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"rate-test-model-c75140-kserve-self-signed-certs\""
Jul 23 00:57:15.343097 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.343080    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c75140\"/\"openshift-service-ca.crt\""
Jul 23 00:57:15.352042 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.352022    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:57:15.367015 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.366964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbwc\" (UniqueName: \"kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.367137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.367029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.367137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.367052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.367137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.367092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.367255 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.367211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.367312 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.367270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468110 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbwc\" (UniqueName: \"kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468110 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468329 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.468862 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.469005 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.469005 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.468915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.470769 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.470736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.471111 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.471093    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.476366 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.476343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbwc\" (UniqueName: \"kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc\") pod \"rate-test-model-c75140-kserve-7c49889f-hfl5b\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") " pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.649225 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.649135    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:15.793003 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:15.792956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:57:15.794448 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:57:15.794420    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b9d071_7a66_4dd5_ae58_84705e715236.slice/crio-09323dca5daf55ed180ab231c34bd8e5f7260dff91e3d36a5c87f8a5e145d234 WatchSource:0}: Error finding container 09323dca5daf55ed180ab231c34bd8e5f7260dff91e3d36a5c87f8a5e145d234: Status 404 returned error can't find the container with id 09323dca5daf55ed180ab231c34bd8e5f7260dff91e3d36a5c87f8a5e145d234
Jul 23 00:57:16.149216 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:16.149183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" event={"ID":"b8b9d071-7a66-4dd5-ae58-84705e715236","Type":"ContainerStarted","Data":"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"}
Jul 23 00:57:16.149216 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:16.149225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" event={"ID":"b8b9d071-7a66-4dd5-ae58-84705e715236","Type":"ContainerStarted","Data":"09323dca5daf55ed180ab231c34bd8e5f7260dff91e3d36a5c87f8a5e145d234"}
Jul 23 00:57:16.167783 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:16.167733    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" podStartSLOduration=1.1677181970000001 podStartE2EDuration="1.167718197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:57:16.167424768 +0000 UTC m=+2971.020994441" watchObservedRunningTime="2026-07-23 00:57:16.167718197 +0000 UTC m=+2971.021287869"
Jul 23 00:57:18.536105 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:18.536069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:57:21.391541 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.391476    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" podUID="e6763842-91f3-4610-a7a3-9e18c819bcd1" containerName="main" containerID="cri-o://fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb" gracePeriod=45
Jul 23 00:57:21.650729 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.650670    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:57:21.725026 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.724999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d64sc\" (UniqueName: \"kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725194 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725356 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725217    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:57:21.725356 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs\") pod \"e6763842-91f3-4610-a7a3-9e18c819bcd1\" (UID: \"e6763842-91f3-4610-a7a3-9e18c819bcd1\") "
Jul 23 00:57:21.725468 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home" (OuterVolumeSpecName: "home") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:57:21.725468 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:57:21.725580 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725569    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:21.725634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725589    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:21.725634 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.725597    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:21.727441 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.727415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:57:21.727666 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.727642    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc" (OuterVolumeSpecName: "kube-api-access-d64sc") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "kube-api-access-d64sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:57:21.727763 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.727746    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm" (OuterVolumeSpecName: "dshm") pod "e6763842-91f3-4610-a7a3-9e18c819bcd1" (UID: "e6763842-91f3-4610-a7a3-9e18c819bcd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:57:21.826304 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.826271    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64sc\" (UniqueName: \"kubernetes.io/projected/e6763842-91f3-4610-a7a3-9e18c819bcd1-kube-api-access-d64sc\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:21.826304 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.826302    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6763842-91f3-4610-a7a3-9e18c819bcd1-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:21.826304 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:21.826314    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6763842-91f3-4610-a7a3-9e18c819bcd1-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:57:22.180569 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.180532    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6763842-91f3-4610-a7a3-9e18c819bcd1" containerID="fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb" exitCode=0
Jul 23 00:57:22.180761 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.180591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"
Jul 23 00:57:22.180761 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.180585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" event={"ID":"e6763842-91f3-4610-a7a3-9e18c819bcd1","Type":"ContainerDied","Data":"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"}
Jul 23 00:57:22.180761 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.180705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x" event={"ID":"e6763842-91f3-4610-a7a3-9e18c819bcd1","Type":"ContainerDied","Data":"26299416885b5046bab31bdce61051c515395215588f511ae09b2597d33cbd99"}
Jul 23 00:57:22.180761 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.180728    2565 scope.go:117] "RemoveContainer" containerID="fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"
Jul 23 00:57:22.190088 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.190072    2565 scope.go:117] "RemoveContainer" containerID="fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"
Jul 23 00:57:22.190349 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:57:22.190333    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb\": container with ID starting with fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb not found: ID does not exist" containerID="fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"
Jul 23 00:57:22.190402 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.190356    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb"} err="failed to get container status \"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb\": rpc error: code = NotFound desc = could not find container \"fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb\": container with ID starting with fbc3396bf33e038617760889048aaab7f27ae387b31730e0feb067380ddda7bb not found: ID does not exist"
Jul 23 00:57:22.203127 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.203107    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:57:22.210263 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:22.210243    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d29ed6/sub-test-model-d29ed6-kserve-5579b5b6cf-j8s8x"]
Jul 23 00:57:23.725459 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:23.725420    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6763842-91f3-4610-a7a3-9e18c819bcd1" path="/var/lib/kubelet/pods/e6763842-91f3-4610-a7a3-9e18c819bcd1/volumes"
Jul 23 00:57:25.649716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:25.649680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:25.649716 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:25.649724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:25.661770 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:25.661741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:26.218692 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:26.218660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:57:47.907369 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:47.907340    2565 scope.go:117] "RemoveContainer" containerID="78f6f385543521faa864c5c6233f1cc205f7da5dc60bf9fb39330d3bbe1b5295"
Jul 23 00:57:57.827263 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:57:57.827224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:58:16.335083 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:16.335050    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:58:31.633210 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:31.633172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:58:38.143918 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:38.143838    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:58:43.414617 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.414577    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906452528"]
Jul 23 00:58:43.415172 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.415153    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6763842-91f3-4610-a7a3-9e18c819bcd1" containerName="main"
Jul 23 00:58:43.415251 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.415174    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6763842-91f3-4610-a7a3-9e18c819bcd1" containerName="main"
Jul 23 00:58:43.415304 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.415277    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6763842-91f3-4610-a7a3-9e18c819bcd1" containerName="main"
Jul 23 00:58:43.418069 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.418047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:43.420737 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.420718    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wjlfl\""
Jul 23 00:58:43.425864 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.425840    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906452528"]
Jul 23 00:58:43.449112 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.449081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28nl\" (UniqueName: \"kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl\") pod \"test-curl-7852-140045906452528\" (UID: \"be7bae06-a5cf-4163-b5ff-0f32df6d79e3\") " pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:43.550133 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.550096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28nl\" (UniqueName: \"kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl\") pod \"test-curl-7852-140045906452528\" (UID: \"be7bae06-a5cf-4163-b5ff-0f32df6d79e3\") " pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:43.560172 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.560128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28nl\" (UniqueName: \"kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl\") pod \"test-curl-7852-140045906452528\" (UID: \"be7bae06-a5cf-4163-b5ff-0f32df6d79e3\") " pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:43.728050 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.727977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:43.865208 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.865181    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906452528"]
Jul 23 00:58:43.869884 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:43.869858    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 00:58:44.587516 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:44.587490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906452528" event={"ID":"be7bae06-a5cf-4163-b5ff-0f32df6d79e3","Type":"ContainerStarted","Data":"d5d3d745065d2b1a118c72c27ddcee0798efa2bf507f999d8369094fb0acab39"}
Jul 23 00:58:45.594585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:45.594550    2565 generic.go:356] "Generic (PLEG): container finished" podID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" containerID="eadb7b9b87be9cbdca99de7a4aa5f65096140dd91a9469a2264150f8b9472d9d" exitCode=0
Jul 23 00:58:45.594982 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:45.594630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906452528" event={"ID":"be7bae06-a5cf-4163-b5ff-0f32df6d79e3","Type":"ContainerDied","Data":"eadb7b9b87be9cbdca99de7a4aa5f65096140dd91a9469a2264150f8b9472d9d"}
Jul 23 00:58:46.732532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.732511    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:46.776273 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.776249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b28nl\" (UniqueName: \"kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl\") pod \"be7bae06-a5cf-4163-b5ff-0f32df6d79e3\" (UID: \"be7bae06-a5cf-4163-b5ff-0f32df6d79e3\") "
Jul 23 00:58:46.778434 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.778410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl" (OuterVolumeSpecName: "kube-api-access-b28nl") pod "be7bae06-a5cf-4163-b5ff-0f32df6d79e3" (UID: "be7bae06-a5cf-4163-b5ff-0f32df6d79e3"). InnerVolumeSpecName "kube-api-access-b28nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:58:46.780556 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.780532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906452528_be7bae06-a5cf-4163-b5ff-0f32df6d79e3/test-curl-7852-140045906452528/0.log"
Jul 23 00:58:46.828393 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.828366    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906452528"]
Jul 23 00:58:46.835485 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.835448    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906452528"]
Jul 23 00:58:46.877094 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:46.877014    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b28nl\" (UniqueName: \"kubernetes.io/projected/be7bae06-a5cf-4163-b5ff-0f32df6d79e3-kube-api-access-b28nl\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:47.606502 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:47.606470    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:47.606678 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:47.606514    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d3d745065d2b1a118c72c27ddcee0798efa2bf507f999d8369094fb0acab39"
Jul 23 00:58:47.616841 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:58:47.616813    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140045906452528\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" pod="opendatahub/test-curl-7852-140045906452528"
Jul 23 00:58:47.726443 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:47.726409    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" path="/var/lib/kubelet/pods/be7bae06-a5cf-4163-b5ff-0f32df6d79e3/volumes"
Jul 23 00:58:50.326483 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:50.326451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:58:52.450165 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.450129    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:52.450550 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.450493    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" containerName="test-curl-7852-140045906452528"
Jul 23 00:58:52.450550 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.450506    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" containerName="test-curl-7852-140045906452528"
Jul 23 00:58:52.450627 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.450586    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7bae06-a5cf-4163-b5ff-0f32df6d79e3" containerName="test-curl-7852-140045906452528"
Jul 23 00:58:52.453817 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.453801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:52.456585 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.456562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wjlfl\""
Jul 23 00:58:52.458730 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.458709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:52.527917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.527877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfdw\" (UniqueName: \"kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw\") pod \"test-curl-7852-140045906544656\" (UID: \"0936a066-a81e-40ed-9e71-e4cea3d58eb2\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:52.629209 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.629174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfdw\" (UniqueName: \"kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw\") pod \"test-curl-7852-140045906544656\" (UID: \"0936a066-a81e-40ed-9e71-e4cea3d58eb2\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:52.641273 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.641248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfdw\" (UniqueName: \"kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw\") pod \"test-curl-7852-140045906544656\" (UID: \"0936a066-a81e-40ed-9e71-e4cea3d58eb2\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:52.764248 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.764211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:52.905486 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:52.905455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:52.908649 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:58:52.908623    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0936a066_a81e_40ed_9e71_e4cea3d58eb2.slice/crio-8d9705fc06f528e51d1b97a9c2cbc8983e9f8349e2e04a5252038a08103cbc46 WatchSource:0}: Error finding container 8d9705fc06f528e51d1b97a9c2cbc8983e9f8349e2e04a5252038a08103cbc46: Status 404 returned error can't find the container with id 8d9705fc06f528e51d1b97a9c2cbc8983e9f8349e2e04a5252038a08103cbc46
Jul 23 00:58:53.164277 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.164195    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" podUID="b8b9d071-7a66-4dd5-ae58-84705e715236" containerName="main" containerID="cri-o://191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48" gracePeriod=45
Jul 23 00:58:53.412997 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.412958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:58:53.537065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbbwc\" (UniqueName: \"kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537232    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537258    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm\") pod \"b8b9d071-7a66-4dd5-ae58-84705e715236\" (UID: \"b8b9d071-7a66-4dd5-ae58-84705e715236\") "
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537461    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:58:53.537533 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537489    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home" (OuterVolumeSpecName: "home") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:58:53.537872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537599    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-model-cache\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.537872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537616    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-home\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.537872 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.537642    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-tmp-dir\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.539230 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.539210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm" (OuterVolumeSpecName: "dshm") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:58:53.539274 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.539223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 00:58:53.539274 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.539263    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc" (OuterVolumeSpecName: "kube-api-access-mbbwc") pod "b8b9d071-7a66-4dd5-ae58-84705e715236" (UID: "b8b9d071-7a66-4dd5-ae58-84705e715236"). InnerVolumeSpecName "kube-api-access-mbbwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:58:53.635254 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.635221    2565 generic.go:356] "Generic (PLEG): container finished" podID="b8b9d071-7a66-4dd5-ae58-84705e715236" containerID="191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48" exitCode=0
Jul 23 00:58:53.635405 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.635268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"
Jul 23 00:58:53.635405 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.635307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" event={"ID":"b8b9d071-7a66-4dd5-ae58-84705e715236","Type":"ContainerDied","Data":"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"}
Jul 23 00:58:53.635405 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.635349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b" event={"ID":"b8b9d071-7a66-4dd5-ae58-84705e715236","Type":"ContainerDied","Data":"09323dca5daf55ed180ab231c34bd8e5f7260dff91e3d36a5c87f8a5e145d234"}
Jul 23 00:58:53.635405 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.635368    2565 scope.go:117] "RemoveContainer" containerID="191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"
Jul 23 00:58:53.636915 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.636890    2565 generic.go:356] "Generic (PLEG): container finished" podID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" containerID="b157848a00da507b162270428b817eff455056f285c9e692f2b15c7aaf0d053f" exitCode=0
Jul 23 00:58:53.637033 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.636928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906544656" event={"ID":"0936a066-a81e-40ed-9e71-e4cea3d58eb2","Type":"ContainerDied","Data":"b157848a00da507b162270428b817eff455056f285c9e692f2b15c7aaf0d053f"}
Jul 23 00:58:53.637033 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.636959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906544656" event={"ID":"0936a066-a81e-40ed-9e71-e4cea3d58eb2","Type":"ContainerStarted","Data":"8d9705fc06f528e51d1b97a9c2cbc8983e9f8349e2e04a5252038a08103cbc46"}
Jul 23 00:58:53.638307 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.638287    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbwc\" (UniqueName: \"kubernetes.io/projected/b8b9d071-7a66-4dd5-ae58-84705e715236-kube-api-access-mbbwc\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.638354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.638311    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b9d071-7a66-4dd5-ae58-84705e715236-tls-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.638354 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.638320    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b9d071-7a66-4dd5-ae58-84705e715236-dshm\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:53.644131 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.644115    2565 scope.go:117] "RemoveContainer" containerID="191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"
Jul 23 00:58:53.644386 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:58:53.644369    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48\": container with ID starting with 191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48 not found: ID does not exist" containerID="191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"
Jul 23 00:58:53.644435 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.644394    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48"} err="failed to get container status \"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48\": rpc error: code = NotFound desc = could not find container \"191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48\": container with ID starting with 191d804b6979e01c098cb031725700619276bd988a46afda5042b650c1ee9c48 not found: ID does not exist"
Jul 23 00:58:53.671471 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.671437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:58:53.678421 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.678396    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c75140/rate-test-model-c75140-kserve-7c49889f-hfl5b"]
Jul 23 00:58:53.726580 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:53.726547    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b9d071-7a66-4dd5-ae58-84705e715236" path="/var/lib/kubelet/pods/b8b9d071-7a66-4dd5-ae58-84705e715236/volumes"
Jul 23 00:58:54.782060 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.782032    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:54.803894 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.803873    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906544656_0936a066-a81e-40ed-9e71-e4cea3d58eb2/test-curl-7852-140045906544656/0.log"
Jul 23 00:58:54.825888 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.825862    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:54.831886 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.831865    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:54.848544 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.848522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gfdw\" (UniqueName: \"kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw\") pod \"0936a066-a81e-40ed-9e71-e4cea3d58eb2\" (UID: \"0936a066-a81e-40ed-9e71-e4cea3d58eb2\") "
Jul 23 00:58:54.850468 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.850446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw" (OuterVolumeSpecName: "kube-api-access-4gfdw") pod "0936a066-a81e-40ed-9e71-e4cea3d58eb2" (UID: "0936a066-a81e-40ed-9e71-e4cea3d58eb2"). InnerVolumeSpecName "kube-api-access-4gfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:58:54.949929 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.949845    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfdw\" (UniqueName: \"kubernetes.io/projected/0936a066-a81e-40ed-9e71-e4cea3d58eb2-kube-api-access-4gfdw\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:54.964019 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.963989    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:54.964341 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b9d071-7a66-4dd5-ae58-84705e715236" containerName="main"
Jul 23 00:58:54.964388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964343    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b9d071-7a66-4dd5-ae58-84705e715236" containerName="main"
Jul 23 00:58:54.964388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964353    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:54.964388 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964360    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:54.964486 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964427    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b9d071-7a66-4dd5-ae58-84705e715236" containerName="main"
Jul 23 00:58:54.964486 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.964436    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:54.967408 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.967391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:54.972764 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:54.972743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:55.051171 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.051141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxc7r\" (UniqueName: \"kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r\") pod \"test-curl-7852-140045906544656\" (UID: \"1032ffa9-4a81-4542-b294-30cba05b371d\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:55.151818 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.151784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxc7r\" (UniqueName: \"kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r\") pod \"test-curl-7852-140045906544656\" (UID: \"1032ffa9-4a81-4542-b294-30cba05b371d\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:55.160637 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.160615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxc7r\" (UniqueName: \"kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r\") pod \"test-curl-7852-140045906544656\" (UID: \"1032ffa9-4a81-4542-b294-30cba05b371d\") " pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:55.277530 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.277495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:55.409152 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.409126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:55.410943 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:58:55.410901    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1032ffa9_4a81_4542_b294_30cba05b371d.slice/crio-834dfe10ddee912182e7f6c98f8f253707b5ec31ba533f93ae2dea8ed3d7f3d5 WatchSource:0}: Error finding container 834dfe10ddee912182e7f6c98f8f253707b5ec31ba533f93ae2dea8ed3d7f3d5: Status 404 returned error can't find the container with id 834dfe10ddee912182e7f6c98f8f253707b5ec31ba533f93ae2dea8ed3d7f3d5
Jul 23 00:58:55.649447 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.649370    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d9705fc06f528e51d1b97a9c2cbc8983e9f8349e2e04a5252038a08103cbc46"
Jul 23 00:58:55.649447 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.649372    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:55.650632 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.650606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906544656" event={"ID":"1032ffa9-4a81-4542-b294-30cba05b371d","Type":"ContainerStarted","Data":"834dfe10ddee912182e7f6c98f8f253707b5ec31ba533f93ae2dea8ed3d7f3d5"}
Jul 23 00:58:55.662452 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.662422    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7852-140045906544656" oldPodUID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" podUID="1032ffa9-4a81-4542-b294-30cba05b371d"
Jul 23 00:58:55.726615 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:55.726588    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0936a066-a81e-40ed-9e71-e4cea3d58eb2" path="/var/lib/kubelet/pods/0936a066-a81e-40ed-9e71-e4cea3d58eb2/volumes"
Jul 23 00:58:56.656913 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:56.656878    2565 generic.go:356] "Generic (PLEG): container finished" podID="1032ffa9-4a81-4542-b294-30cba05b371d" containerID="0a7e247e89c29e84fdeb4ae053d637b16633da7f0fa533a80ba3b786cd616ae4" exitCode=0
Jul 23 00:58:56.657309 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:56.656988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7852-140045906544656" event={"ID":"1032ffa9-4a81-4542-b294-30cba05b371d","Type":"ContainerDied","Data":"0a7e247e89c29e84fdeb4ae053d637b16633da7f0fa533a80ba3b786cd616ae4"}
Jul 23 00:58:57.795408 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.795384    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:57.816532 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.816506    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7852-140045906544656_1032ffa9-4a81-4542-b294-30cba05b371d/test-curl-7852-140045906544656/0.log"
Jul 23 00:58:57.839332 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.839303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:57.842714 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.842692    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7852-140045906544656"]
Jul 23 00:58:57.875183 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.875155    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxc7r\" (UniqueName: \"kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r\") pod \"1032ffa9-4a81-4542-b294-30cba05b371d\" (UID: \"1032ffa9-4a81-4542-b294-30cba05b371d\") "
Jul 23 00:58:57.877382 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.877356    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r" (OuterVolumeSpecName: "kube-api-access-zxc7r") pod "1032ffa9-4a81-4542-b294-30cba05b371d" (UID: "1032ffa9-4a81-4542-b294-30cba05b371d"). InnerVolumeSpecName "kube-api-access-zxc7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:58:57.976519 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:57.976441    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxc7r\" (UniqueName: \"kubernetes.io/projected/1032ffa9-4a81-4542-b294-30cba05b371d-kube-api-access-zxc7r\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:58:58.032349 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.032317    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"]
Jul 23 00:58:58.032762 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.032744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:58.032845 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.032765    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:58.032895 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.032856    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" containerName="test-curl-7852-140045906544656"
Jul 23 00:58:58.035899 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.035880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:58:58.038684 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.038664    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 00:58:58.038774 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.038696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 00:58:58.043595 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.043572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"]
Jul 23 00:58:58.077420 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.077382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb\" (UID: \"084ac998-50f4-4164-ac0c-16d9ebf7b336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:58:58.177851 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.177808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb\" (UID: \"084ac998-50f4-4164-ac0c-16d9ebf7b336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:58:58.178554 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.178534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb\" (UID: \"084ac998-50f4-4164-ac0c-16d9ebf7b336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:58:58.348572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.348531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:58:58.484237 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.483866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"]
Jul 23 00:58:58.486600 ip-10-0-141-133 kubenswrapper[2565]: W0723 00:58:58.486572    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084ac998_50f4_4164_ac0c_16d9ebf7b336.slice/crio-0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd WatchSource:0}: Error finding container 0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd: Status 404 returned error can't find the container with id 0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd
Jul 23 00:58:58.669505 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.669420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb" event={"ID":"084ac998-50f4-4164-ac0c-16d9ebf7b336","Type":"ContainerStarted","Data":"0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd"}
Jul 23 00:58:58.670834 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.670810    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:58.670834 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:58.670820    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="834dfe10ddee912182e7f6c98f8f253707b5ec31ba533f93ae2dea8ed3d7f3d5"
Jul 23 00:58:58.685778 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:58:58.685747    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140045906544656\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:59.678404 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:59.678371    2565 generic.go:356] "Generic (PLEG): container finished" podID="084ac998-50f4-4164-ac0c-16d9ebf7b336" containerID="3dc0b2c6cbd5419dd39c68f7c422d81d8ca3bf3487e434d82eae08143ed9e21b" exitCode=0
Jul 23 00:58:59.678784 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:59.678454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb" event={"ID":"084ac998-50f4-4164-ac0c-16d9ebf7b336","Type":"ContainerDied","Data":"3dc0b2c6cbd5419dd39c68f7c422d81d8ca3bf3487e434d82eae08143ed9e21b"}
Jul 23 00:58:59.681066 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:58:59.681044    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7852-140045906544656\" is forbidden: User \"system:node:ip-10-0-141-133.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-133.ec2.internal' and this object" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" pod="opendatahub/test-curl-7852-140045906544656"
Jul 23 00:58:59.726065 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:58:59.726038    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1032ffa9-4a81-4542-b294-30cba05b371d" path="/var/lib/kubelet/pods/1032ffa9-4a81-4542-b294-30cba05b371d/volumes"
Jul 23 00:59:00.830159 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:00.830136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:59:00.901178 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:00.901138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca\") pod \"084ac998-50f4-4164-ac0c-16d9ebf7b336\" (UID: \"084ac998-50f4-4164-ac0c-16d9ebf7b336\") "
Jul 23 00:59:00.901485 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:00.901463    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "084ac998-50f4-4164-ac0c-16d9ebf7b336" (UID: "084ac998-50f4-4164-ac0c-16d9ebf7b336"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:59:01.002809 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:01.002775    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/084ac998-50f4-4164-ac0c-16d9ebf7b336-maas-api-service-ca\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:01.689955 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:01.689916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb" event={"ID":"084ac998-50f4-4164-ac0c-16d9ebf7b336","Type":"ContainerDied","Data":"0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd"}
Jul 23 00:59:01.690221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:01.689994    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c0a64be0b347cc94981f2bb95839f3757d20ffef0032efbf3271c690b94a9fd"
Jul 23 00:59:01.690221 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:01.689936    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"
Jul 23 00:59:11.518205 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:11.518167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:59:13.159452 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:13.159416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"]
Jul 23 00:59:13.170456 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:13.170427    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c75140-xfsmb"]
Jul 23 00:59:13.731923 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:13.731883    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="084ac998-50f4-4164-ac0c-16d9ebf7b336" path="/var/lib/kubelet/pods/084ac998-50f4-4164-ac0c-16d9ebf7b336/volumes"
Jul 23 00:59:20.034271 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:20.034238    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:59:20.034874 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:20.034447    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy" containerID="cri-o://3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4" gracePeriod=30
Jul 23 00:59:25.294113 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.294088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:59:25.421224 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421224 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421188    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421224 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421224 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421226    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421256    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421359    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket\") pod \"c122d01d-018f-4dfe-b888-e2e693969cb1\" (UID: \"c122d01d-018f-4dfe-b888-e2e693969cb1\") "
Jul 23 00:59:25.421572 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421438    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:59:25.421917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421640    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:59:25.421917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421700    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-certs\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.421917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 00:59:25.421917 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.421830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:59:25.422146 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.422115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:59:25.423652 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.423619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 00:59:25.423652 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.423645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt" (OuterVolumeSpecName: "kube-api-access-4ljqt") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "kube-api-access-4ljqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:59:25.423803 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.423665    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 00:59:25.423910 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.423891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c122d01d-018f-4dfe-b888-e2e693969cb1" (UID: "c122d01d-018f-4dfe-b888-e2e693969cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 00:59:25.522509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522474    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-credential-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522503    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-kube-api-access-4ljqt\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522509 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522516    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-token\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522529    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c122d01d-018f-4dfe-b888-e2e693969cb1-istiod-ca-cert\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522539    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-data\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522550    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-workload-socket\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522586    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-podinfo\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.522745 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.522598    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c122d01d-018f-4dfe-b888-e2e693969cb1-istio-envoy\") on node \"ip-10-0-141-133.ec2.internal\" DevicePath \"\""
Jul 23 00:59:25.803646 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.803612    2565 generic.go:356] "Generic (PLEG): container finished" podID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerID="3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4" exitCode=0
Jul 23 00:59:25.803815 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.803660    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"
Jul 23 00:59:25.803815 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.803699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerDied","Data":"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"}
Jul 23 00:59:25.803815 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.803751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch" event={"ID":"c122d01d-018f-4dfe-b888-e2e693969cb1","Type":"ContainerDied","Data":"58284acd61949a74610eb485399b2a5d4bf7b6be87b16c74cb441600cd9fa62b"}
Jul 23 00:59:25.803815 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.803775    2565 scope.go:117] "RemoveContainer" containerID="3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"
Jul 23 00:59:25.814504 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.814486    2565 scope.go:117] "RemoveContainer" containerID="92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"
Jul 23 00:59:25.822499 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.822475    2565 scope.go:117] "RemoveContainer" containerID="3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"
Jul 23 00:59:25.822770 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:59:25.822748    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4\": container with ID starting with 3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4 not found: ID does not exist" containerID="3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"
Jul 23 00:59:25.822828 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.822781    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4"} err="failed to get container status \"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4\": rpc error: code = NotFound desc = could not find container \"3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4\": container with ID starting with 3d1a7ab152b57b3764344c1857f8abc7150501eeb64270fa8eec7dafaa1a01c4 not found: ID does not exist"
Jul 23 00:59:25.822828 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.822804    2565 scope.go:117] "RemoveContainer" containerID="92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"
Jul 23 00:59:25.823050 ip-10-0-141-133 kubenswrapper[2565]: E0723 00:59:25.823031    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9\": container with ID starting with 92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9 not found: ID does not exist" containerID="92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"
Jul 23 00:59:25.823137 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.823055    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9"} err="failed to get container status \"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9\": rpc error: code = NotFound desc = could not find container \"92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9\": container with ID starting with 92357bc6943b6c5302560fccf8280468a0e651db4d3865f799fff66c4828fcb9 not found: ID does not exist"
Jul 23 00:59:25.823208 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.823192    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:59:25.832831 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:25.832801    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c75140-openshift-default-6dd5f7c87-6lrch"]
Jul 23 00:59:27.725243 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:27.725202    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" path="/var/lib/kubelet/pods/c122d01d-018f-4dfe-b888-e2e693969cb1/volumes"
Jul 23 00:59:32.939100 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:32.939067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bcvbx"]
Jul 23 00:59:48.326903 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.326857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hdq46_a6e29054-2d36-4542-9fb1-8dc19acc3f5b/manager/0.log"
Jul 23 00:59:48.442269 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.442237    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ksgsr_c64a6008-4724-4601-a990-30b4f28eac42/manager/0.log"
Jul 23 00:59:48.567809 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.567783    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fccc86db5-2ccj2_0d6840e7-577e-4d3c-a391-81e4d76b85d3/manager/0.log"
Jul 23 00:59:48.686835 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.686768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2478h_d6e13d01-01ee-4585-81bd-e88c40221074/server/0.log"
Jul 23 00:59:48.809698 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.809675    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mq54c_1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3/manager/0.log"
Jul 23 00:59:48.931331 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.931309    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk_cb6c2601-95ba-4c7d-a712-067396648a9d/copy-manifests/0.log"
Jul 23 00:59:48.938480 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:48.938413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk_cb6c2601-95ba-4c7d-a712-067396648a9d/manager/0.log"
Jul 23 00:59:51.510610 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:51.510579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xfjz_2981629d-879c-47ea-8b36-ed6429d2d2a1/manager/0.log"
Jul 23 00:59:51.626651 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:51.626623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xq8nm_fa017948-914d-4a1b-ab3c-ef8b8e38244d/manager/0.log"
Jul 23 00:59:51.738123 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:51.738100    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rg5zn_7b310e1e-c3a8-48da-8804-c2c733991bfc/kuadrant-console-plugin/0.log"
Jul 23 00:59:51.852829 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:51.852761    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4k85j_85243c52-df37-473e-9894-edaaf5aaf91d/registry-server/0.log"
Jul 23 00:59:51.973370 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:51.973343    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jgq8m_a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b/manager/0.log"
Jul 23 00:59:52.080406 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:52.080380    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bcvbx_48113520-45cc-4a33-840b-733c0a348870/limitador/0.log"
Jul 23 00:59:52.813789 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:52.813763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-hn8hx_91d287c6-23be-4296-bf56-f8ef83a3272b/kube-auth-proxy/0.log"
Jul 23 00:59:53.063102 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:53.063070    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/1.log"
Jul 23 00:59:53.193958 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:53.193865    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-jdqt4_ba58cb79-afcb-471f-88b2-edd213181d52/payload-pre-processing/0.log"
Jul 23 00:59:53.834999 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:53.834954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bzkmr_131a67ff-3ea8-4ab0-8cb4-265710c8c3f9/main/0.log"
Jul 23 00:59:53.953498 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:53.953456    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xbktw_35c01219-0888-4e0e-a9a7-56fe45ded8a8/main/0.log"
Jul 23 00:59:54.464272 ip-10-0-141-133 kubenswrapper[2565]: I0723 00:59:54.464231    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vkvcg_29faca5d-19c5-43ee-bad1-f53d6132290a/main/0.log"
Jul 23 01:00:00.143265 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143231    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"]
Jul 23 01:00:00.143658 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143605    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.143658 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143617    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.143658 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143626    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="084ac998-50f4-4164-ac0c-16d9ebf7b336" containerName="revoke-keys"
Jul 23 01:00:00.143658 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143631    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="084ac998-50f4-4164-ac0c-16d9ebf7b336" containerName="revoke-keys"
Jul 23 01:00:00.143799 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143703    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.143799 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.143717    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="084ac998-50f4-4164-ac0c-16d9ebf7b336" containerName="revoke-keys"
Jul 23 01:00:00.146790 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.146775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"
Jul 23 01:00:00.150069 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.150044    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w8tk2\""
Jul 23 01:00:00.155652 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.155630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"]
Jul 23 01:00:00.157383 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.157356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"
Jul 23 01:00:00.320849 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.320820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"]
Jul 23 01:00:00.323318 ip-10-0-141-133 kubenswrapper[2565]: W0723 01:00:00.323289    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4522dd_958a_4742_b88a_5bf802eacb10.slice/crio-7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682 WatchSource:0}: Error finding container 7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682: Status 404 returned error can't find the container with id 7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682
Jul 23 01:00:00.366181 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.363409    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtxqj/must-gather-sxl56"]
Jul 23 01:00:00.366181 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.364248    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.366181 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.364269    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.366181 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.364429    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c122d01d-018f-4dfe-b888-e2e693969cb1" containerName="istio-proxy"
Jul 23 01:00:00.369512 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.369483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.372877 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.372855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtxqj\"/\"default-dockercfg-t6rqd\""
Jul 23 01:00:00.373265 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.373240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"openshift-service-ca.crt\""
Jul 23 01:00:00.373364 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.373339    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtxqj\"/\"kube-root-ca.crt\""
Jul 23 01:00:00.379358 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.379336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/must-gather-sxl56"]
Jul 23 01:00:00.520903 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.520864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6de19fc-c8b0-4851-a739-6acb3cf57421-must-gather-output\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.521113 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.520915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7sv\" (UniqueName: \"kubernetes.io/projected/a6de19fc-c8b0-4851-a739-6acb3cf57421-kube-api-access-pz7sv\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.621666 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.621633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6de19fc-c8b0-4851-a739-6acb3cf57421-must-gather-output\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.621823 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.621694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7sv\" (UniqueName: \"kubernetes.io/projected/a6de19fc-c8b0-4851-a739-6acb3cf57421-kube-api-access-pz7sv\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.622000 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.621956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a6de19fc-c8b0-4851-a739-6acb3cf57421-must-gather-output\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.629766 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.629744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7sv\" (UniqueName: \"kubernetes.io/projected/a6de19fc-c8b0-4851-a739-6acb3cf57421-kube-api-access-pz7sv\") pod \"must-gather-sxl56\" (UID: \"a6de19fc-c8b0-4851-a739-6acb3cf57421\") " pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.679424 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.679396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/must-gather-sxl56"
Jul 23 01:00:00.809601 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.809576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/must-gather-sxl56"]
Jul 23 01:00:00.811719 ip-10-0-141-133 kubenswrapper[2565]: W0723 01:00:00.811693    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6de19fc_c8b0_4851_a739_6acb3cf57421.slice/crio-2110dc260fd44bcc6230f784d06e9de1b964873a70cdf470100437bb47deb14a WatchSource:0}: Error finding container 2110dc260fd44bcc6230f784d06e9de1b964873a70cdf470100437bb47deb14a: Status 404 returned error can't find the container with id 2110dc260fd44bcc6230f784d06e9de1b964873a70cdf470100437bb47deb14a
Jul 23 01:00:00.974851 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.974813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx" event={"ID":"4d4522dd-958a-4742-b88a-5bf802eacb10","Type":"ContainerStarted","Data":"735129259e7d71444d34005c91cf549e89ee96a53d0404c3601f201f745fe291"}
Jul 23 01:00:00.975067 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.974883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx" event={"ID":"4d4522dd-958a-4742-b88a-5bf802eacb10","Type":"ContainerStarted","Data":"7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682"}
Jul 23 01:00:00.975999 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.975955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/must-gather-sxl56" event={"ID":"a6de19fc-c8b0-4851-a739-6acb3cf57421","Type":"ContainerStarted","Data":"2110dc260fd44bcc6230f784d06e9de1b964873a70cdf470100437bb47deb14a"}
Jul 23 01:00:00.991974 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:00.991926    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx" podStartSLOduration=0.991911982 podStartE2EDuration="991.911982ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:00.990453756 +0000 UTC m=+3135.844023429" watchObservedRunningTime="2026-07-23 01:00:00.991911982 +0000 UTC m=+3135.845481655"
Jul 23 01:00:01.982249 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:01.982212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/must-gather-sxl56" event={"ID":"a6de19fc-c8b0-4851-a739-6acb3cf57421","Type":"ContainerStarted","Data":"fd4692d1235f044d715b7a726fc6afea06c12ab401e5c847834acbe8492a1b13"}
Jul 23 01:00:01.982651 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:01.982281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/must-gather-sxl56" event={"ID":"a6de19fc-c8b0-4851-a739-6acb3cf57421","Type":"ContainerStarted","Data":"4e5d2d90da6791aa676463f91d23b5728c4aa4b153dc5138ab2d97683c93cf64"}
Jul 23 01:00:01.983928 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:01.983905    2565 generic.go:356] "Generic (PLEG): container finished" podID="4d4522dd-958a-4742-b88a-5bf802eacb10" containerID="735129259e7d71444d34005c91cf549e89ee96a53d0404c3601f201f745fe291" exitCode=0
Jul 23 01:00:01.984057 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:01.983934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx" event={"ID":"4d4522dd-958a-4742-b88a-5bf802eacb10","Type":"ContainerDied","Data":"735129259e7d71444d34005c91cf549e89ee96a53d0404c3601f201f745fe291"}
Jul 23 01:00:02.001544 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:02.001477    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtxqj/must-gather-sxl56" podStartSLOduration=1.030813268 podStartE2EDuration="2.001457353s" totalImagesPullingTime="970.644085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:02.00077852 +0000 UTC m=+3136.854348222" watchObservedRunningTime="2026-07-23 01:00:02.001457353 +0000 UTC m=+3136.855027027"
Jul 23 01:00:03.151641 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.151611    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"
Jul 23 01:00:03.378435 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.378403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j9hf9_ce30ed35-f91a-40eb-992a-ed17e7a1bd30/global-pull-secret-syncer/0.log"
Jul 23 01:00:03.586830 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.586794    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v5btk_b6c68d68-0526-45e1-966a-883bb02511be/konnectivity-agent/0.log"
Jul 23 01:00:03.657543 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.657462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-133.ec2.internal_642dc77bb1e95d4ad1afe2c58591ff7f/haproxy/0.log"
Jul 23 01:00:03.999735 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.999709    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx"
Jul 23 01:00:04.000635 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:03.999705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-kt2wx" event={"ID":"4d4522dd-958a-4742-b88a-5bf802eacb10","Type":"ContainerDied","Data":"7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682"}
Jul 23 01:00:04.000635 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:04.000590    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e93e3f2682751fa4d119e9ef101b03815621a042397760f3d31fa88537a2682"
Jul 23 01:00:08.971086 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:08.970922    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5xfjz_2981629d-879c-47ea-8b36-ed6429d2d2a1/manager/0.log"
Jul 23 01:00:09.002455 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:09.002415    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xq8nm_fa017948-914d-4a1b-ab3c-ef8b8e38244d/manager/0.log"
Jul 23 01:00:09.047043 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:09.047017    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rg5zn_7b310e1e-c3a8-48da-8804-c2c733991bfc/kuadrant-console-plugin/0.log"
Jul 23 01:00:09.127021 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:09.126959    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4k85j_85243c52-df37-473e-9894-edaaf5aaf91d/registry-server/0.log"
Jul 23 01:00:09.304294 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:09.304252    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jgq8m_a7eb9da3-9f3b-49cd-8b22-b93b7d5d1c6b/manager/0.log"
Jul 23 01:00:09.328421 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:09.328386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bcvbx_48113520-45cc-4a33-840b-733c0a348870/limitador/0.log"
Jul 23 01:00:11.592259 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:11.592222    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cxgz_ad736d8f-ce73-4e1e-843f-377295ac59f1/node-exporter/0.log"
Jul 23 01:00:11.615512 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:11.615485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cxgz_ad736d8f-ce73-4e1e-843f-377295ac59f1/kube-rbac-proxy/0.log"
Jul 23 01:00:11.645492 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:11.645465    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cxgz_ad736d8f-ce73-4e1e-843f-377295ac59f1/init-textfile/0.log"
Jul 23 01:00:13.493076 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.493030    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"]
Jul 23 01:00:13.494287 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.494259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4522dd-958a-4742-b88a-5bf802eacb10" containerName="cleanup"
Jul 23 01:00:13.494444 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.494431    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4522dd-958a-4742-b88a-5bf802eacb10" containerName="cleanup"
Jul 23 01:00:13.494671 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.494656    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4522dd-958a-4742-b88a-5bf802eacb10" containerName="cleanup"
Jul 23 01:00:13.499762 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.499736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.512026 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.511994    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"]
Jul 23 01:00:13.559305 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.559268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-proc\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.559495 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.559318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-lib-modules\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.559495 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.559427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-sys\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.559495 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.559470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-podres\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.559616 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.559524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2j76\" (UniqueName: \"kubernetes.io/projected/887764e0-a0f9-43b3-a5c9-714eea6c43b2-kube-api-access-h2j76\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.603993 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.603944    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-5kjdk_3101d750-8df5-4897-9428-b552c33a6c74/networking-console-plugin/0.log"
Jul 23 01:00:13.660185 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-sys\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-podres\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2j76\" (UniqueName: \"kubernetes.io/projected/887764e0-a0f9-43b3-a5c9-714eea6c43b2-kube-api-access-h2j76\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-sys\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-proc\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-proc\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660604 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-podres\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660604 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-lib-modules\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.660604 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.660498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/887764e0-a0f9-43b3-a5c9-714eea6c43b2-lib-modules\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.670378 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.670352    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2j76\" (UniqueName: \"kubernetes.io/projected/887764e0-a0f9-43b3-a5c9-714eea6c43b2-kube-api-access-h2j76\") pod \"perf-node-gather-daemonset-c5mrg\" (UID: \"887764e0-a0f9-43b3-a5c9-714eea6c43b2\") " pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.812643 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.812552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:13.997585 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:13.997491    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"]
Jul 23 01:00:14.009890 ip-10-0-141-133 kubenswrapper[2565]: W0723 01:00:14.009861    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod887764e0_a0f9_43b3_a5c9_714eea6c43b2.slice/crio-56e8f93fe28e6d1d66128eb894bd996064615ad740454ddb3afef10a5f745201 WatchSource:0}: Error finding container 56e8f93fe28e6d1d66128eb894bd996064615ad740454ddb3afef10a5f745201: Status 404 returned error can't find the container with id 56e8f93fe28e6d1d66128eb894bd996064615ad740454ddb3afef10a5f745201
Jul 23 01:00:14.073159 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:14.071752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg" event={"ID":"887764e0-a0f9-43b3-a5c9-714eea6c43b2","Type":"ContainerStarted","Data":"56e8f93fe28e6d1d66128eb894bd996064615ad740454ddb3afef10a5f745201"}
Jul 23 01:00:15.080235 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:15.080190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg" event={"ID":"887764e0-a0f9-43b3-a5c9-714eea6c43b2","Type":"ContainerStarted","Data":"5ed3cd25989573b733241f8f71ecb79c73e463c846fa1b57504104113f66cecb"}
Jul 23 01:00:15.080651 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:15.080345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:15.101430 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:15.101374    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg" podStartSLOduration=2.10135464 podStartE2EDuration="2.10135464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:15.100222439 +0000 UTC m=+3149.953792113" watchObservedRunningTime="2026-07-23 01:00:15.10135464 +0000 UTC m=+3149.954924314"
Jul 23 01:00:16.546286 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:16.546256    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8cbw_21cfc68c-5751-42f7-a018-383e03338cd7/dns/0.log"
Jul 23 01:00:16.569621 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:16.569596    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8cbw_21cfc68c-5751-42f7-a018-383e03338cd7/kube-rbac-proxy/0.log"
Jul 23 01:00:16.624125 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:16.624093    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwthh_629cb05f-3479-4f5b-a410-1c32ec4d52d1/dns-node-resolver/0.log"
Jul 23 01:00:17.250918 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:17.250889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7p5th_d283f102-2e7d-4d9f-a493-aed1369cd78d/node-ca/0.log"
Jul 23 01:00:18.438983 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:18.438950    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-hn8hx_91d287c6-23be-4296-bf56-f8ef83a3272b/kube-auth-proxy/0.log"
Jul 23 01:00:18.495160 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:18.495118    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/0.log"
Jul 23 01:00:18.530379 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:18.530351    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/1.log"
Jul 23 01:00:18.563659 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:18.563634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-jdqt4_ba58cb79-afcb-471f-88b2-edd213181d52/payload-pre-processing/0.log"
Jul 23 01:00:19.344121 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:19.344092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-thxht_a98c84a3-e7bf-451f-84bb-d0a6914fb9d6/serve-healthcheck-canary/0.log"
Jul 23 01:00:19.842685 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:19.842661    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56xk7_481ac944-2252-4ac5-ba03-4d641edf2a2a/kube-rbac-proxy/0.log"
Jul 23 01:00:19.866879 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:19.866849    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56xk7_481ac944-2252-4ac5-ba03-4d641edf2a2a/exporter/0.log"
Jul 23 01:00:19.891258 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:19.891229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56xk7_481ac944-2252-4ac5-ba03-4d641edf2a2a/extractor/0.log"
Jul 23 01:00:21.097918 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:21.097892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtxqj/perf-node-gather-daemonset-c5mrg"
Jul 23 01:00:22.260929 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.260899    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hdq46_a6e29054-2d36-4542-9fb1-8dc19acc3f5b/manager/0.log"
Jul 23 01:00:22.523491 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.523458    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ksgsr_c64a6008-4724-4601-a990-30b4f28eac42/manager/0.log"
Jul 23 01:00:22.687212 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.686468    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fccc86db5-2ccj2_0d6840e7-577e-4d3c-a391-81e4d76b85d3/manager/0.log"
Jul 23 01:00:22.711297 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.711270    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2478h_d6e13d01-01ee-4585-81bd-e88c40221074/server/0.log"
Jul 23 01:00:22.802328 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.802299    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mq54c_1d5eee87-e866-453d-ae6f-fe8dfa1d6ef3/manager/0.log"
Jul 23 01:00:22.832107 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.832076    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk_cb6c2601-95ba-4c7d-a712-067396648a9d/manager/0.log"
Jul 23 01:00:22.858541 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:22.858516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f65bfdcb7-k7hrk_cb6c2601-95ba-4c7d-a712-067396648a9d/copy-manifests/0.log"
Jul 23 01:00:23.687626 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:23.687603    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-hn8hx_91d287c6-23be-4296-bf56-f8ef83a3272b/kube-auth-proxy/0.log"
Jul 23 01:00:23.724570 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:23.724537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/0.log"
Jul 23 01:00:23.760122 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:23.760093    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/1.log"
Jul 23 01:00:23.782650 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:23.782621    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-jdqt4_ba58cb79-afcb-471f-88b2-edd213181d52/payload-pre-processing/0.log"
Jul 23 01:00:25.142905 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:25.142876    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7447dc7bcf-hn8hx_91d287c6-23be-4296-bf56-f8ef83a3272b/kube-auth-proxy/0.log"
Jul 23 01:00:25.189497 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:25.189459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/0.log"
Jul 23 01:00:25.221678 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:25.221571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6fnhl_8f310acc-20bd-4048-bf4f-a08ace8d187d/istio-proxy/1.log"
Jul 23 01:00:25.249010 ip-10-0-141-133 kubenswrapper[2565]: I0723 01:00:25.248292    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-jdqt4_ba58cb79-afcb-471f-88b2-edd213181d52/payload-pre-processing/0.log"