Jul 23 11:08:12.787364 ip-10-0-129-186 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 11:08:12.787379 ip-10-0-129-186 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 11:08:12.787387 ip-10-0-129-186 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 11:08:12.787657 ip-10-0-129-186 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 11:08:22.871766 ip-10-0-129-186 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 23 11:08:22.871782 ip-10-0-129-186 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6581bc69fbe64e85bf8ef45c1c9fbbfb --
Jul 23 11:10:26.051377 ip-10-0-129-186 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 11:10:26.442188 ip-10-0-129-186 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:10:26.442188 ip-10-0-129-186 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 11:10:26.442188 ip-10-0-129-186 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:10:26.442188 ip-10-0-129-186 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 11:10:26.442188 ip-10-0-129-186 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:10:26.443013 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.442900    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 11:10:26.448365 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448346    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.448365 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448363    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.448365 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448368    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.448365 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448372    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448377    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448380    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448384    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448387    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448393    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448397    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448400    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448404    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448407    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448411    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448415    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448419    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448422    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448433    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448437    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448440    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448444    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448448    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.448616 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448453    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448457    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448461    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448465    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448469    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448473    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448476    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448480    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448484    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448488    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448493    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448497    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448500    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448507    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448512    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448517    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448522    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448526    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448531    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448535    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.449422 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448539    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448544    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448549    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448553    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448558    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448562    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448566    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448571    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448575    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448579    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448583    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448587    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448591    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448595    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448600    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448604    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448609    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448613    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448616    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448625    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.450248 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448631    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448636    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448641    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448645    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448649    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448655    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448660    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448666    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448671    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448676    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448680    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448684    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448688    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448692    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448697    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448703    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448707    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448711    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448715    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448722    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.451066 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448729    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448734    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448740    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448744    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448748    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448752    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448756    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448761    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448765    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448776    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448780    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448784    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448788    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448795    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448801    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448805    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.448809    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.449990    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450001    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.451914 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450006    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450011    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450015    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450021    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450025    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450029    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450034    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450038    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450042    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450047    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450052    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450057    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450062    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450066    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450072    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450078    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450083    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450088    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450115    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.452744 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450119    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450124    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450128    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450132    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450137    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450141    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450145    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450149    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450153    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450157    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450161    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450165    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450169    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450174    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450178    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450182    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450188    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450194    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450199    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450203    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.453522 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450208    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450213    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450218    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450224    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450228    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450232    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450236    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450240    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450244    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450247    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450251    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450255    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450258    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450262    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450266    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450271    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450275    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450279    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450283    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.454026 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450288    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450294    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450300    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450304    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450309    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450314    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450318    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450323    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450327    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450331    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450335    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450343    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450347    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450351    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450355    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450359    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450363    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450370    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450374    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450379    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.454577 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450383    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450387    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450391    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450395    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450399    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450403    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450407    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450411    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450415    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450419    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450424    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450428    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450432    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450436    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450440    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450444    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450448    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450452    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.450457    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451184    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.455330 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451193    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451198    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451202    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451209    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451214    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451218    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451222    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451226    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451230    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451242    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451246    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451250    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451255    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451259    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451265    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451271    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451276    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451281    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451285    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451290    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.455801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451294    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451299    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451304    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451308    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451312    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451316    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451320    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451324    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451328    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451332    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451337    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451342    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451346    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451350    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451354    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451360    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451364    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451368    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451372    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451376    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.456323 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451380    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451384    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451399    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451403    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451407    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451411    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451415    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451419    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451423    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451427    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451431    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451435    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451441    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451444    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451449    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451453    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451457    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451461    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451465    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.456894 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451471    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451476    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451481    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451485    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451490    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451494    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451498    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451502    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451508    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451512    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451517    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451520    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451524    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451529    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451533    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451537    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451549    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451553    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451557    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451561    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.457445 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451565    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451569    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451574    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451579    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451584    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451589    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451593    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451597    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451601    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451605    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451609    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451613    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451617    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451622    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451626    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451630    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451634    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451638    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.451642    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451747    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 11:10:26.457917 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451759    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451773    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451780    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451786    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451791    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451797    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451804    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451809    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451814    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451827    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451833    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451838    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451843    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451848    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451852    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451857    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451862    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451871    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451876    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451881    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451886    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451891    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451897    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 11:10:26.458423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451904    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451908    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451913    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451917    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451922    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451926    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451931    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451935    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451941    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451947    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451951    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451956    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451960    2583 flags.go:64] FLAG: --enable-server="true"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451965    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451976    2583 flags.go:64] FLAG: --event-burst="100"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451981    2583 flags.go:64] FLAG: --event-qps="50"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451985    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451990    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.451995    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452009    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452013    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452018    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452022    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452027    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452031    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 11:10:26.458989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452036    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452040    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452045    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452049    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452054    2583 flags.go:64] FLAG: --feature-gates=""
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452059    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452064    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452070    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452075    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452080    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452085    2583 flags.go:64] FLAG: --help="false"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452109    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452115    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452119    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452124    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452130    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452135    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452140    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452145    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452150    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452155    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452159    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452164    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452169    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 23 11:10:26.459660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452173    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452178    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452183    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452195    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452200    2583 flags.go:64] FLAG: --lock-file=""
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452205    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452209    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452214    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452223    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452228    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452233    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452237    2583 flags.go:64] FLAG: --logging-format="text"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452241    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452247    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452251    2583 flags.go:64] FLAG: --manifest-url=""
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452257    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452264    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452269    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452275    2583 flags.go:64] FLAG: --max-pods="110"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452280    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452284    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452288    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452293    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452297    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452303    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 11:10:26.460338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452307    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452318    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452322    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452327    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452332    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452336    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452344    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452348    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452354    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452359    2583 flags.go:64] FLAG: --port="10250"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452374    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452387    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03c480bee325b6d37"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452393    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452397    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452402    2583 flags.go:64] FLAG: --register-node="true"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452407    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452412    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452417    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452422    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452447    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452454    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452459    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452464    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452470    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452475    2583 flags.go:64] FLAG: --runonce="false"
Jul 23 11:10:26.460934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452479    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452484    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452489    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452493    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452498    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452502    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452507    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452512    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452516    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452521    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452526    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452530    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452535    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452540    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452544    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452552    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452557    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452561    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452571    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452583    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452588    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452593    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452597    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452602    2583 flags.go:64] FLAG: --v="2"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452608    2583 flags.go:64] FLAG: --version="false"
Jul 23 11:10:26.461688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452614    2583 flags.go:64] FLAG: --vmodule=""
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452619    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.452624    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452889    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452896    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452904    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452909    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452913    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452917    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452922    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452926    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452932    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452936    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452941    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452945    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452949    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452953    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452957    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452961    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452965    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.462389 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452970    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452974    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452978    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452982    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452986    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452990    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.452994    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453006    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453011    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453015    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453019    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453023    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453027    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453031    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453034    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453038    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453043    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453048    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453052    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453056    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.462911 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453060    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453065    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453069    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453075    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453079    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453083    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453087    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453113    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453117    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453121    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453125    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453128    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453133    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453137    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453140    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453144    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453148    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453152    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453158    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453164    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.463424 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453177    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453182    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453186    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453190    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453194    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453198    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453203    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453207    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453211    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453217    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453221    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453225    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453229    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453233    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453237    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453242    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453247    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453251    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453255    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.463889 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453260    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453264    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453268    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453272    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453278    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453284    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453290    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453295    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453300    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453304    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453309    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453313    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453317    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453321    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453333    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453338    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453342    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453346    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.464377 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453350    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453354    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453358    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453362    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.453370    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.453973    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.463497    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.463514    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.463534    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.463541    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463615    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463621    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463624    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463627    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463629    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.464816 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463632    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463635    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463638    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463640    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463643    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463646    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463649    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463652    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463654    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463657    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463659    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463662    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463665    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463667    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463670    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463673    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463676    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463679    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463682    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463684    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.465216 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463687    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463689    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463692    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463694    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463697    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463700    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463702    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463705    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463707    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463710    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463712    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463715    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463718    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463720    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463723    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463725    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463728    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463730    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463733    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.465699 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463735    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463737    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463740    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463743    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463746    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463748    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463751    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463753    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463756    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463759    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463762    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463766    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463769    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463772    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463775    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463778    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463781    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463784    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463787    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463789    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.466237 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463792    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463795    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463797    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463799    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463802    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463805    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463807    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463810    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463812    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463816    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463819    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463822    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463825    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463828    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463830    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463834    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463837    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463839    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463842    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.466727 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463844    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463847    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463849    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463852    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463854    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463857    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463859    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463862    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463864    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463867    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463871    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463874    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463877    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463879    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463882    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.467202 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.463884    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.463889    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464026    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464032    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464035    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464037    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464041    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464045    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464049    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464052    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464055    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464057    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464060    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464063    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464066    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:10:26.467556 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464069    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464071    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464074    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464076    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464079    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464081    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464084    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464087    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464108    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464112    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464114    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464116    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464119    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464121    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464124    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464126    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464129    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464131    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464134    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464137    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:10:26.467931 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464139    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464143    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464146    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464149    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464151    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464154    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464157    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464159    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464162    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464165    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464167    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464170    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464175    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464178    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464182    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464184    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464187    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464189    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464192    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:10:26.468430 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464194    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464196    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464199    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464201    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464204    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464206    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464209    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464212    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464214    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464216    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464219    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464221    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464224    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464226    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464229    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464231    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464234    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464236    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464238    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464241    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:10:26.468903 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464243    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464246    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464248    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464250    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464253    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464255    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464258    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464261    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464264    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464266    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464268    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464271    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464273    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464276    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464278    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464280    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464283    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464285    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464287    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464290    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:10:26.469398 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464292    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464295    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464297    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464300    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464302    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464305    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:26.464307    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.464313    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.464424    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.466394    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.467860    2583 server.go:1019] "Starting client certificate rotation"
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.467953    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 11:10:26.469871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.468744    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 11:10:26.470865 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.470851    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 11:10:26.474357 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.474341    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 11:10:26.483937 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.483920    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 23 11:10:26.488489 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.488472    2583 log.go:25] "Validated CRI v1 image API"
Jul 23 11:10:26.491368 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.491353    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 11:10:26.494199 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.494165    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 92f6237b-143f-4b92-ba17-8baa604bd887:/dev/nvme0n1p3 99eb859f-8ec5-4c89-bb04-8dcc6f446109:/dev/nvme0n1p4]
Jul 23 11:10:26.494247 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.494201    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 11:10:26.500854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.500832    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 11:10:26.501932 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.501825    2583 manager.go:216] Machine: {Timestamp:2026-07-23 11:10:26.500333365 +0000 UTC m=+0.333141589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101202 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a5a66aaca14b0b7e5d0cf0939aa40 SystemUUID:ec2a5a66-aaca-14b0-b7e5-d0cf0939aa40 BootID:6581bc69-fbe6-4e85-bf8e-f45c1c9fbbfb Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:20:c3:f3:71:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:20:c3:f3:71:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:06:95:46:e0:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 11:10:26.501932 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.501928    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 11:10:26.502042 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.502014    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 11:10:26.504851 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.504826    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 11:10:26.504984 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.504853    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-186.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 11:10:26.505495 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.505486    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 11:10:26.505527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.505498    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 11:10:26.505527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.505511    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 11:10:26.506215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.506205    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 11:10:26.507293 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.507283    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 11:10:26.511057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511046    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 11:10:26.511192 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511181    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 11:10:26.511233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511215    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 11:10:26.511233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511226    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 11:10:26.511294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511236    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 11:10:26.511294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511248    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 23 11:10:26.511294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511257    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 11:10:26.511613 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511594    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 11:10:26.511704 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.511623    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 11:10:26.513774 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.513751    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 11:10:26.515752 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.515739    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 11:10:26.515794 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.515769    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 11:10:26.516857 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516844    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516863    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516869    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516875    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516884    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516900    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 11:10:26.516907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516909    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 11:10:26.517074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516914    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 11:10:26.517074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516921    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 11:10:26.517074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516926    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 11:10:26.517074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516934    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 11:10:26.517074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.516941    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 11:10:26.517813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.517802    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 11:10:26.517845 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.517814    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 11:10:26.520799 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.520762    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-186.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 11:10:26.520888 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.520812    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 11:10:26.520888 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.520857    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 11:10:26.521612 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.521592    2583 server.go:1268] "Started kubelet"
Jul 23 11:10:26.521699 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.521643    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 11:10:26.521763 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.521664    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 11:10:26.521811 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.521789    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 11:10:26.522466 ip-10-0-129-186 systemd[1]: Started Kubernetes Kubelet.
Jul 23 11:10:26.523442 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.523420    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 11:10:26.523685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.523663    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 11:10:26.528086 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.528057    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 11:10:26.529109 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.528085    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-186.ec2.internal.18c4e6b71f24a892  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-186.ec2.internal,UID:ip-10-0-129-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-186.ec2.internal,},FirstTimestamp:2026-07-23 11:10:26.521458834 +0000 UTC m=+0.354267058,LastTimestamp:2026-07-23 11:10:26.521458834 +0000 UTC m=+0.354267058,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-186.ec2.internal,}"
Jul 23 11:10:26.529632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.529618    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 11:10:26.529701 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.529630    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 11:10:26.529862 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.529841    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.530170 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530149    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 11:10:26.530170 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530156    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 11:10:26.530324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530176    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 11:10:26.530324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530303    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 11:10:26.530324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530312    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 11:10:26.530601 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530584    2583 factory.go:55] Registering systemd factory
Jul 23 11:10:26.530601 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.530604    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 23 11:10:26.531067 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531053    2583 factory.go:153] Registering CRI-O factory
Jul 23 11:10:26.531067 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531066    2583 factory.go:223] Registration of the crio container factory successfully
Jul 23 11:10:26.531245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531127    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 11:10:26.531245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531145    2583 factory.go:103] Registering Raw factory
Jul 23 11:10:26.531245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531156    2583 manager.go:1196] Started watching for new ooms in manager
Jul 23 11:10:26.531463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.531449    2583 manager.go:349] Starting recovery of all containers
Jul 23 11:10:26.532319 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.532295    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nhx7t"
Jul 23 11:10:26.536707 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.536667    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 11:10:26.536830 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.536807    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 11:10:26.541437 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.541411    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nhx7t"
Jul 23 11:10:26.544416 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.544399    2583 manager.go:354] Recovery completed
Jul 23 11:10:26.545766 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.545737    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 23 11:10:26.548821 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.548803    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.550642 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.550622    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.550708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.550653    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.550708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.550662    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.551195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551179    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 11:10:26.551195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551195    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 11:10:26.551296 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551210    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 11:10:26.551296 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551185    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.551888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551871    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.551958 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551905    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.551958 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.551922    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.552555 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.552495    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-186.ec2.internal.18c4e6b720e1ea1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-186.ec2.internal,UID:ip-10-0-129-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-186.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-186.ec2.internal,},FirstTimestamp:2026-07-23 11:10:26.55063913 +0000 UTC m=+0.383447357,LastTimestamp:2026-07-23 11:10:26.55063913 +0000 UTC m=+0.383447357,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-186.ec2.internal,}"
Jul 23 11:10:26.553348 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.553336    2583 policy_none.go:49] "None policy: Start"
Jul 23 11:10:26.553390 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.553352    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 11:10:26.553390 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.553362    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 11:10:26.555330 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.555317    2583 policy_none.go:47] "Start"
Jul 23 11:10:26.594431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594269    2583 manager.go:337] "Starting Device Plugin manager"
Jul 23 11:10:26.594543 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.594459    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 11:10:26.594543 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594474    2583 server.go:87] "Starting device plugin registration server"
Jul 23 11:10:26.594693 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594680    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 11:10:26.594735 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594696    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 11:10:26.594824 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594808    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 11:10:26.594917 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594905    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 11:10:26.594967 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.594918    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 11:10:26.595432 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.595411    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 11:10:26.595507 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.595469    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.695599 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.695530    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.697500 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.696452    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.697500 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.696486    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.697500 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.696500    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.697500 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.696543    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.705431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.705415    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.705510 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.705438    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-186.ec2.internal\": node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.731710 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.731690    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.737265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.737230    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 11:10:26.738716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.738694    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 11:10:26.738800 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.738738    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 11:10:26.738800 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.738761    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 11:10:26.738800 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.738796    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 11:10:26.743242 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.743224    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:10:26.831837 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.831793    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.838933 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.838894    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"]
Jul 23 11:10:26.838994 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.838983    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.840045 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.840028    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.840133 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.840057    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.840133 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.840107    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.842226 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.842213    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.842372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.842357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.842418 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.842389    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.843155 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.843140    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.843202 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.843169    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.843202 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.843182    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.844082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.844069    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.844153 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.844144    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.844200 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.844160    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.845134 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.845120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.845195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.845146    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:10:26.845906 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.845889    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:10:26.845997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.845915    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:10:26.845997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.845925    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:10:26.862734 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.862707    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-186.ec2.internal\" not found" node="ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.866737 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.866720    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-186.ec2.internal\" not found" node="ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.931928 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:26.931888    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:26.932126 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.931937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.932126 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.931969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:26.932126 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:26.931994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5ee178cb91c3b2f8f69b5e375b061e5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"d5ee178cb91c3b2f8f69b5e375b061e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032658 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.032579    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.032658 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5ee178cb91c3b2f8f69b5e375b061e5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"d5ee178cb91c3b2f8f69b5e375b061e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d1440d8ab02edb51626f8bb4bbcbf86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal\" (UID: \"2d1440d8ab02edb51626f8bb4bbcbf86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.032808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.032749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d5ee178cb91c3b2f8f69b5e375b061e5-config\") pod \"kube-apiserver-proxy-ip-10-0-129-186.ec2.internal\" (UID: \"d5ee178cb91c3b2f8f69b5e375b061e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.132985 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.132947    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.166166 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.166139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.169287 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.169262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.233900 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.233867    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.334413 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.334344    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.434786 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.434744    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.466930 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.466903    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 11:10:27.467360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.467055    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 11:10:27.535721 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.535688    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.548722 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.548693    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:05:27 +0000 UTC" deadline="2028-03-11 08:02:15.72459132 +0000 UTC"
Jul 23 11:10:27.548722 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.548720    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14324h51m48.175874314s"
Jul 23 11:10:27.636548 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.636518    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.737232 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.737197    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.820601 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.820574    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:10:27.837726 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:27.837696    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-186.ec2.internal\" not found"
Jul 23 11:10:27.896640 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:27.896599    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1440d8ab02edb51626f8bb4bbcbf86.slice/crio-b5643678b2d70837963372a2b9272ae1ee7e9fbb7aca7f4f2a78beb1dd655213 WatchSource:0}: Error finding container b5643678b2d70837963372a2b9272ae1ee7e9fbb7aca7f4f2a78beb1dd655213: Status 404 returned error can't find the container with id b5643678b2d70837963372a2b9272ae1ee7e9fbb7aca7f4f2a78beb1dd655213
Jul 23 11:10:27.901020 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.901002    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:10:27.917986 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.917959    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:10:27.925119 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.925101    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:10:27.930475 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.930458    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.964826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.964794    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 11:10:27.966062 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.966045    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal"
Jul 23 11:10:27.980175 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:27.980141    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 11:10:28.513078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.513032    2583 apiserver.go:52] "Watching apiserver"
Jul 23 11:10:28.521373 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.521352    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 11:10:28.521779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.521757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal","openshift-multus/multus-4jcc8","openshift-multus/multus-additional-cni-plugins-5p4hs","openshift-multus/network-metrics-daemon-dv8dx","openshift-network-diagnostics/network-check-target-mm4c2","openshift-ovn-kubernetes/ovnkube-node-hbxgn","kube-system/global-pull-secret-syncer-mts9v","kube-system/konnectivity-agent-7nbsm","openshift-cluster-node-tuning-operator/tuned-hxs7t","openshift-dns/node-resolver-tvvkn","openshift-image-registry/node-ca-shvrq","openshift-network-operator/iptables-alerter-9r5q5"]
Jul 23 11:10:28.525769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.525756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:28.525854 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.525835    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:28.528185 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.528164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.530272 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.530242    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 11:10:28.530483 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.530464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.532836 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.532815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.533095 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.533069    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 11:10:28.533388 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.533371    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ftpzq\""
Jul 23 11:10:28.533455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.533405    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.533514 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.533476    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.534135 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.534118    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7n6f\""
Jul 23 11:10:28.535309 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.535245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 11:10:28.535381 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.535342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.535859 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.535839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.535953 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.535889    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 11:10:28.536155 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.536135    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.536295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.536278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.536364 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.536277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 11:10:28.536426 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.536391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2n6rh\""
Jul 23 11:10:28.536551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.536534    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.537888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.537872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.540499 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.540481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.541706 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.541676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 11:10:28.541854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.541764    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2jx2j\""
Jul 23 11:10:28.541854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.541817    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.542455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7n6\" (UniqueName: \"kubernetes.io/projected/1075af97-b1ff-40bc-8742-d35953d18a85-kube-api-access-cq7n6\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.542455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-iptables-alerter-script\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.542455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-host-slash\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.542455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-kube-api-access-7vrq4\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.542730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:28.542730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1075af97-b1ff-40bc-8742-d35953d18a85-host\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.542730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1075af97-b1ff-40bc-8742-d35953d18a85-serviceca\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.542730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542592    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 11:10:28.542730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542642    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 11:10:28.543082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.542994    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 11:10:28.543082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.543082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lxvpr\""
Jul 23 11:10:28.543370 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543306    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 11:10:28.543804 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543782    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.543910 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543848    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 11:10:28.543976 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.543937    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q2ztt\""
Jul 23 11:10:28.544217 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.544158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.545739 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.545712    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.546381 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.546355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.546740 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.546716    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.546962 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.546946    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.547035 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.546948    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w6xbc\""
Jul 23 11:10:28.549082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.549063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.549891 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.549875    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 11:10:28.550501 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.550485    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 11:10:28.550596 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.550555    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkn2r\""
Jul 23 11:10:28.550661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.550602    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 11:10:28.551016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.551000    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 11:10:28.551765 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.551748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.551875 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.551856    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:28.551943 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.551890    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8p45v\""
Jul 23 11:10:28.552124 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.551987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 11:10:28.552365 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.552345    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 11:10:28.554428 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.554409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.554502 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.554465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:28.574480 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.574461    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mqv78"
Jul 23 11:10:28.581441 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.581423    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mqv78"
Jul 23 11:10:28.630955 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.630929    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 11:10:28.642781 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-systemd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.642861 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-var-lib-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.642861 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e655262a-c02f-4320-b3c9-6b4c22b446b0-tmp-dir\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.642861 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-host\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.642960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.642960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-systemd-units\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.642960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.642960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-config\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.643078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.643078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.642993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-k8s-cni-cncf-io\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-kubelet\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-conf-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643078 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-etc-kubernetes\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-iptables-alerter-script\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.643262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.643262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-conf\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.643262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-kubelet\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-kubernetes\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-var-lib-kubelet\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.643463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmws\" (UniqueName: \"kubernetes.io/projected/d377a039-1e9c-41f4-a468-52fcc97e49de-kube-api-access-ncmws\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-kube-api-access-7vrq4\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm42g\" (UniqueName: \"kubernetes.io/projected/e655262a-c02f-4320-b3c9-6b4c22b446b0-kube-api-access-zm42g\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfh2\" (UniqueName: \"kubernetes.io/projected/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-kube-api-access-skfh2\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-multus-certs\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfmz\" (UniqueName: \"kubernetes.io/projected/dd73de0a-9361-4d9e-9486-a4beb65010c9-kube-api-access-hcfmz\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.643673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-os-release\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-etc-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-modprobe-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-iptables-alerter-script\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.643919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-sys-fs\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-cnibin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxp2l\" (UniqueName: \"kubernetes.io/projected/c51760f5-5a51-4204-b88b-10b587e42069-kube-api-access-hxp2l\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.643991    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e655262a-c02f-4320-b3c9-6b4c22b446b0-hosts-file\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-device-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-multus\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7n6\" (UniqueName: \"kubernetes.io/projected/1075af97-b1ff-40bc-8742-d35953d18a85-kube-api-access-cq7n6\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-host-slash\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-netd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-tuned\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-host-slash\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.644245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-os-release\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-dbus\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-cni-binary-copy\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-kubelet-config\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-registration-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-system-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cnibin\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1075af97-b1ff-40bc-8742-d35953d18a85-host\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-netns\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-log-socket\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1075af97-b1ff-40bc-8742-d35953d18a85-host\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysconfig\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-netns\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-bin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-system-cni-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.644793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-node-log\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-env-overrides\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-sys\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfl9\" (UniqueName: \"kubernetes.io/projected/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-kube-api-access-dmfl9\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-ovn\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l2z\" (UniqueName: \"kubernetes.io/projected/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-kube-api-access-44l2z\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-hostroot\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-daemon-config\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/498c5f37-ded0-46f2-8784-4598551995eb-agent-certs\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-bin\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-script-lib\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.644979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-systemd\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-run\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-etc-selinux\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-socket-dir-parent\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1075af97-b1ff-40bc-8742-d35953d18a85-serviceca\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.645606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-slash\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.646181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-lib-modules\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.646181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-tmp\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.646181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-socket-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.646181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/498c5f37-ded0-46f2-8784-4598551995eb-konnectivity-ca\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.646181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.645504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1075af97-b1ff-40bc-8742-d35953d18a85-serviceca\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.652164 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.652138    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 11:10:28.653312 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.653248    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:28.653312 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.653275    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:28.653312 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.653289    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:28.653516 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.653359    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:29.153331982 +0000 UTC m=+2.986140223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:28.655770 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.655747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7n6\" (UniqueName: \"kubernetes.io/projected/1075af97-b1ff-40bc-8742-d35953d18a85-kube-api-access-cq7n6\") pod \"node-ca-shvrq\" (UID: \"1075af97-b1ff-40bc-8742-d35953d18a85\") " pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.655860 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.655750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrq4\" (UniqueName: \"kubernetes.io/projected/cc18943f-4684-43f8-b3ec-9fcfd0e7bc26-kube-api-access-7vrq4\") pod \"iptables-alerter-9r5q5\" (UID: \"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26\") " pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.681425 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.681397    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:10:28.744785 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.744739    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"2d1440d8ab02edb51626f8bb4bbcbf86","Type":"ContainerStarted","Data":"b5643678b2d70837963372a2b9272ae1ee7e9fbb7aca7f4f2a78beb1dd655213"}
Jul 23 11:10:28.746027 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-registration-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.746111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-system-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cnibin\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-netns\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-log-socket\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-registration-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-netns\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysconfig\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-system-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-log-socket\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cnibin\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysconfig\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.746249 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-netns\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-bin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-netns\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-system-cni-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-bin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-node-log\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-system-cni-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-node-log\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-env-overrides\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-sys\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfl9\" (UniqueName: \"kubernetes.io/projected/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-kube-api-access-dmfl9\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-ovn\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746490    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-sys\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l2z\" (UniqueName: \"kubernetes.io/projected/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-kube-api-access-44l2z\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.746545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-hostroot\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-ovn\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-daemon-config\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/498c5f37-ded0-46f2-8784-4598551995eb-agent-certs\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-bin\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-script-lib\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-systemd\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-run\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-etc-selinux\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-bin\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-socket-dir-parent\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-slash\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-socket-dir-parent\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-lib-modules\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-run\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-tmp\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-hostroot\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-socket-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.747609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/498c5f37-ded0-46f2-8784-4598551995eb-konnectivity-ca\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-lib-modules\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-systemd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-slash\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-var-lib-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-env-overrides\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.746991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e655262a-c02f-4320-b3c9-6b4c22b446b0-tmp-dir\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-host\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-systemd\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-socket-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-systemd-units\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-etc-selinux\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-config\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.748632 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-k8s-cni-cncf-io\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-kubelet\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-conf-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-etc-kubernetes\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-conf\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-daemon-config\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e655262a-c02f-4320-b3c9-6b4c22b446b0-tmp-dir\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.747579    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/498c5f37-ded0-46f2-8784-4598551995eb-konnectivity-ca\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.747642    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:10:29.247624189 +0000 UTC m=+3.080432417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-host\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-script-lib\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.749620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-systemd-units\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-kubelet\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-conf\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-conf-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-kubernetes\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-etc-kubernetes\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-kubelet\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-var-lib-kubelet\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-kubernetes\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-var-lib-kubelet\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmws\" (UniqueName: \"kubernetes.io/projected/d377a039-1e9c-41f4-a468-52fcc97e49de-kube-api-access-ncmws\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm42g\" (UniqueName: \"kubernetes.io/projected/e655262a-c02f-4320-b3c9-6b4c22b446b0-kube-api-access-zm42g\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.747988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfh2\" (UniqueName: \"kubernetes.io/projected/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-kube-api-access-skfh2\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748031    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-k8s-cni-cncf-io\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.750524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-multus-certs\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfmz\" (UniqueName: \"kubernetes.io/projected/dd73de0a-9361-4d9e-9486-a4beb65010c9-kube-api-access-hcfmz\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-os-release\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-etc-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-var-lib-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-multus-cni-dir\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-modprobe-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-sys-fs\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-modprobe-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-etc-openvswitch\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-sys-fs\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovnkube-config\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-run-systemd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-cnibin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.751098 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-run-multus-certs\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-kubelet\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-cnibin\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp2l\" (UniqueName: \"kubernetes.io/projected/c51760f5-5a51-4204-b88b-10b587e42069-kube-api-access-hxp2l\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e655262a-c02f-4320-b3c9-6b4c22b446b0-hosts-file\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-device-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-multus\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-netd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e655262a-c02f-4320-b3c9-6b4c22b446b0-hosts-file\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-tuned\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-sysctl-d\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-host-cni-netd\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51760f5-5a51-4204-b88b-10b587e42069-device-dir\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-host-var-lib-cni-multus\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-os-release\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.748978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-os-release\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-dbus\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.751746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-cni-binary-copy\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-kubelet-config\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-dbus\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd73de0a-9361-4d9e-9486-a4beb65010c9-os-release\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70cb2736-a420-4e4d-be73-a9a12d0c3f15-kubelet-config\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.749406    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:28.749554    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:29.249528627 +0000 UTC m=+3.082336842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd73de0a-9361-4d9e-9486-a4beb65010c9-cni-binary-copy\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.749843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-tmp\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.750754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/498c5f37-ded0-46f2-8784-4598551995eb-agent-certs\") pod \"konnectivity-agent-7nbsm\" (UID: \"498c5f37-ded0-46f2-8784-4598551995eb\") " pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.750982    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-etc-tuned\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.752423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.751512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.758571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.758548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfl9\" (UniqueName: \"kubernetes.io/projected/c43c0bf8-f3c5-4b05-8800-c5a7a186703c-kube-api-access-dmfl9\") pod \"multus-additional-cni-plugins-5p4hs\" (UID: \"c43c0bf8-f3c5-4b05-8800-c5a7a186703c\") " pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.762067 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.762041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmws\" (UniqueName: \"kubernetes.io/projected/d377a039-1e9c-41f4-a468-52fcc97e49de-kube-api-access-ncmws\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:28.762523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.762498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm42g\" (UniqueName: \"kubernetes.io/projected/e655262a-c02f-4320-b3c9-6b4c22b446b0-kube-api-access-zm42g\") pod \"node-resolver-tvvkn\" (UID: \"e655262a-c02f-4320-b3c9-6b4c22b446b0\") " pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.763207 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.763161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l2z\" (UniqueName: \"kubernetes.io/projected/9c9061b6-e435-4a9b-92c4-cccb7f5774b2-kube-api-access-44l2z\") pod \"ovnkube-node-hbxgn\" (UID: \"9c9061b6-e435-4a9b-92c4-cccb7f5774b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.763668 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.763648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfh2\" (UniqueName: \"kubernetes.io/projected/d65b3a24-773e-4ac9-a4eb-615ef57a13aa-kube-api-access-skfh2\") pod \"tuned-hxs7t\" (UID: \"d65b3a24-773e-4ac9-a4eb-615ef57a13aa\") " pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.763905 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.763884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfmz\" (UniqueName: \"kubernetes.io/projected/dd73de0a-9361-4d9e-9486-a4beb65010c9-kube-api-access-hcfmz\") pod \"multus-4jcc8\" (UID: \"dd73de0a-9361-4d9e-9486-a4beb65010c9\") " pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.765172 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.765152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxp2l\" (UniqueName: \"kubernetes.io/projected/c51760f5-5a51-4204-b88b-10b587e42069-kube-api-access-hxp2l\") pod \"aws-ebs-csi-driver-node-xh8n8\" (UID: \"c51760f5-5a51-4204-b88b-10b587e42069\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.838399 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.838364    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shvrq"
Jul 23 11:10:28.848295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.848272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9r5q5"
Jul 23 11:10:28.849829 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.849795    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1075af97_b1ff_40bc_8742_d35953d18a85.slice/crio-c78a070c3d1bb6fb6553c8710e7fd65f3200bf8c3093bea772716448cefd8fba WatchSource:0}: Error finding container c78a070c3d1bb6fb6553c8710e7fd65f3200bf8c3093bea772716448cefd8fba: Status 404 returned error can't find the container with id c78a070c3d1bb6fb6553c8710e7fd65f3200bf8c3093bea772716448cefd8fba
Jul 23 11:10:28.856660 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.856640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4jcc8"
Jul 23 11:10:28.858721 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.858693    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc18943f_4684_43f8_b3ec_9fcfd0e7bc26.slice/crio-8d7e1f4fcef7618e415f48766b4fad22532d7ff5cc4044e1dbcbe1430579d4eb WatchSource:0}: Error finding container 8d7e1f4fcef7618e415f48766b4fad22532d7ff5cc4044e1dbcbe1430579d4eb: Status 404 returned error can't find the container with id 8d7e1f4fcef7618e415f48766b4fad22532d7ff5cc4044e1dbcbe1430579d4eb
Jul 23 11:10:28.863517 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.863495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:28.865536 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.865516    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd73de0a_9361_4d9e_9486_a4beb65010c9.slice/crio-6c41cd8f53595fbb0c926d881545f16cb0975ca3c98e6ca920842944946a3649 WatchSource:0}: Error finding container 6c41cd8f53595fbb0c926d881545f16cb0975ca3c98e6ca920842944946a3649: Status 404 returned error can't find the container with id 6c41cd8f53595fbb0c926d881545f16cb0975ca3c98e6ca920842944946a3649
Jul 23 11:10:28.869307 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.869208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5p4hs"
Jul 23 11:10:28.873147 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.873047    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498c5f37_ded0_46f2_8784_4598551995eb.slice/crio-ef4ec179ac6207bcdf7528f328322c9a5d65424d789c8621605ee2a67d606056 WatchSource:0}: Error finding container ef4ec179ac6207bcdf7528f328322c9a5d65424d789c8621605ee2a67d606056: Status 404 returned error can't find the container with id ef4ec179ac6207bcdf7528f328322c9a5d65424d789c8621605ee2a67d606056
Jul 23 11:10:28.874612 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.874598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:28.879451 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.879428    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43c0bf8_f3c5_4b05_8800_c5a7a186703c.slice/crio-a77bd730bdcf9957959699e805b733f9c64fbf62b5f51ea45c4bf4330f29833d WatchSource:0}: Error finding container a77bd730bdcf9957959699e805b733f9c64fbf62b5f51ea45c4bf4330f29833d: Status 404 returned error can't find the container with id a77bd730bdcf9957959699e805b733f9c64fbf62b5f51ea45c4bf4330f29833d
Jul 23 11:10:28.881014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.880995    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tvvkn"
Jul 23 11:10:28.886032 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.886010    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9061b6_e435_4a9b_92c4_cccb7f5774b2.slice/crio-eb88d5192b3aa543b12c2c39ad660ac40b5d6daf564cfd2619abf7fb4072fc6c WatchSource:0}: Error finding container eb88d5192b3aa543b12c2c39ad660ac40b5d6daf564cfd2619abf7fb4072fc6c: Status 404 returned error can't find the container with id eb88d5192b3aa543b12c2c39ad660ac40b5d6daf564cfd2619abf7fb4072fc6c
Jul 23 11:10:28.886806 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.886788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxs7t"
Jul 23 11:10:28.890966 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:28.890952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8"
Jul 23 11:10:28.892162 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.892136    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode655262a_c02f_4320_b3c9_6b4c22b446b0.slice/crio-6c73ab51c2ffdad7c3e82ba0f06ea5f87cfc1796f807eb902040cab147dbefae WatchSource:0}: Error finding container 6c73ab51c2ffdad7c3e82ba0f06ea5f87cfc1796f807eb902040cab147dbefae: Status 404 returned error can't find the container with id 6c73ab51c2ffdad7c3e82ba0f06ea5f87cfc1796f807eb902040cab147dbefae
Jul 23 11:10:28.897333 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.897315    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65b3a24_773e_4ac9_a4eb_615ef57a13aa.slice/crio-b2721366c45f7ea30080410e2a12ee1ae0e235ee632af9e5f3278ad8e78abf5c WatchSource:0}: Error finding container b2721366c45f7ea30080410e2a12ee1ae0e235ee632af9e5f3278ad8e78abf5c: Status 404 returned error can't find the container with id b2721366c45f7ea30080410e2a12ee1ae0e235ee632af9e5f3278ad8e78abf5c
Jul 23 11:10:28.900332 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:28.900314    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51760f5_5a51_4204_b88b_10b587e42069.slice/crio-bcb39e3f8a44f741de1c94284c0e2d5e0ea895d7887b1977913fd168f03737f3 WatchSource:0}: Error finding container bcb39e3f8a44f741de1c94284c0e2d5e0ea895d7887b1977913fd168f03737f3: Status 404 returned error can't find the container with id bcb39e3f8a44f741de1c94284c0e2d5e0ea895d7887b1977913fd168f03737f3
Jul 23 11:10:29.204129 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:10:29.204099    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee178cb91c3b2f8f69b5e375b061e5.slice/crio-b47bbeba0c989025e34fb7c19c216883fec1acb78dd918ef19b35481777a8c20 WatchSource:0}: Error finding container b47bbeba0c989025e34fb7c19c216883fec1acb78dd918ef19b35481777a8c20: Status 404 returned error can't find the container with id b47bbeba0c989025e34fb7c19c216883fec1acb78dd918ef19b35481777a8c20
Jul 23 11:10:29.253909 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.253874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:29.254084 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.253926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:29.254084 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254048    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:29.254084 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254054    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:29.254084 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254068    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:29.254084 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254085    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:29.254344 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.254108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:29.254344 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254185    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:30.254124658 +0000 UTC m=+4.086932874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:29.254344 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254216    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:29.254344 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254245    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:10:30.254218868 +0000 UTC m=+4.087027100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:29.254344 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.254287    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:30.254270867 +0000 UTC m=+4.087079102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:29.584939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.584810    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:05:29 +0000 UTC" deadline="2028-02-06 08:46:20.228145555 +0000 UTC"
Jul 23 11:10:29.584939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.584885    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13509h35m50.64326488s"
Jul 23 11:10:29.740916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.740106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:29.740916 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.740264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:29.740916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.740709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:29.740916 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:29.740811    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:29.749265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.749224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxs7t" event={"ID":"d65b3a24-773e-4ac9-a4eb-615ef57a13aa","Type":"ContainerStarted","Data":"b2721366c45f7ea30080410e2a12ee1ae0e235ee632af9e5f3278ad8e78abf5c"}
Jul 23 11:10:29.751210 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.751143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvvkn" event={"ID":"e655262a-c02f-4320-b3c9-6b4c22b446b0","Type":"ContainerStarted","Data":"6c73ab51c2ffdad7c3e82ba0f06ea5f87cfc1796f807eb902040cab147dbefae"}
Jul 23 11:10:29.755945 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.755861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerStarted","Data":"a77bd730bdcf9957959699e805b733f9c64fbf62b5f51ea45c4bf4330f29833d"}
Jul 23 11:10:29.766624 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.766559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7nbsm" event={"ID":"498c5f37-ded0-46f2-8784-4598551995eb","Type":"ContainerStarted","Data":"ef4ec179ac6207bcdf7528f328322c9a5d65424d789c8621605ee2a67d606056"}
Jul 23 11:10:29.771088 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.771053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jcc8" event={"ID":"dd73de0a-9361-4d9e-9486-a4beb65010c9","Type":"ContainerStarted","Data":"6c41cd8f53595fbb0c926d881545f16cb0975ca3c98e6ca920842944946a3649"}
Jul 23 11:10:29.784299 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.784266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9r5q5" event={"ID":"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26","Type":"ContainerStarted","Data":"8d7e1f4fcef7618e415f48766b4fad22532d7ff5cc4044e1dbcbe1430579d4eb"}
Jul 23 11:10:29.791745 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.791712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8" event={"ID":"c51760f5-5a51-4204-b88b-10b587e42069","Type":"ContainerStarted","Data":"bcb39e3f8a44f741de1c94284c0e2d5e0ea895d7887b1977913fd168f03737f3"}
Jul 23 11:10:29.801942 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.801842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"eb88d5192b3aa543b12c2c39ad660ac40b5d6daf564cfd2619abf7fb4072fc6c"}
Jul 23 11:10:29.816727 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.816696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shvrq" event={"ID":"1075af97-b1ff-40bc-8742-d35953d18a85","Type":"ContainerStarted","Data":"c78a070c3d1bb6fb6553c8710e7fd65f3200bf8c3093bea772716448cefd8fba"}
Jul 23 11:10:29.825942 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:29.825881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" event={"ID":"d5ee178cb91c3b2f8f69b5e375b061e5","Type":"ContainerStarted","Data":"b47bbeba0c989025e34fb7c19c216883fec1acb78dd918ef19b35481777a8c20"}
Jul 23 11:10:30.264829 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.264737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:30.264829 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.264804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:30.265060 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.264847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:30.265060 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265010    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:30.265259 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265093    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:32.265071911 +0000 UTC m=+6.097880130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:30.265576 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265556    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:30.265655 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265619    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:10:32.265601635 +0000 UTC m=+6.098409853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:30.265754 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265738    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:30.265822 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265764    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:30.265822 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265778    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:30.265822 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.265816    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:32.265802934 +0000 UTC m=+6.098611150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:30.585537 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.585436    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:05:29 +0000 UTC" deadline="2028-05-05 06:07:46.89065886 +0000 UTC"
Jul 23 11:10:30.585537 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.585483    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15642h57m16.305181292s"
Jul 23 11:10:30.741707 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:30.741671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:30.741869 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:30.741817    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:31.739791 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:31.739760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:31.740325 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:31.739909    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:31.740325 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:31.740254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:31.740452 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:31.740361    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:32.283040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:32.283100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:32.283161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283306    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283373    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:36.28335341 +0000 UTC m=+10.116161627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283796    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283850    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:10:36.283832581 +0000 UTC m=+10.116640820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283963    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283981    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.283994    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:32.284065 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.284027    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:36.284017715 +0000 UTC m=+10.116825927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:32.743610 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:32.742906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:32.743610 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:32.743139    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:33.740160 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:33.739417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:33.740160 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:33.739558    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:33.740160 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:33.739995    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:33.740160 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:33.740107    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:34.743291 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:34.743262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:34.743764 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:34.743383    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:35.738969 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:35.738936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:35.739149 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:35.738936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:35.739149 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:35.739125    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:35.739319 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:35.739224    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:36.318846 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:36.318798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:36.319391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:36.318861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:36.319391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:36.318906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:36.319391 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319074    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:36.319391 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:44.319125626 +0000 UTC m=+18.151933864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:36.319661 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319550    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:36.319661 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319607    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:10:44.319590579 +0000 UTC m=+18.152398811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:36.319738 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319675    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:36.319738 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319694    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:36.319738 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319706    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:36.319852 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.319744    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:10:44.319735154 +0000 UTC m=+18.152543385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:36.739327 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:36.739235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:36.739507 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:36.739371    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:37.740294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:37.739636    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:37.740294 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:37.739783    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:37.740294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:37.740176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:37.740294 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:37.740261    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:38.739511 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:38.739468    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:38.739701 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:38.739612    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:39.739321 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:39.739249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:39.739738 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:39.739437    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:39.739738 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:39.739489    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:39.739738 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:39.739584    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:40.739234 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:40.739205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:40.739407 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:40.739315    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:41.739107 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:41.739003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:41.739107 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:41.739038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:41.739321 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:41.739162    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:41.739321 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:41.739271    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:42.739037 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:42.739007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:42.739486 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:42.739134    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:43.739993 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:43.739963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:43.740430 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:43.740072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:43.740430 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:43.739958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:43.740430 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:43.740195    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:44.386402 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:44.386367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:44.386588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:44.386435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:44.386588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:44.386473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:44.386588 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386533    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:44.386588 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386589    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386592    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386601    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386615    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386602    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:11:00.386581113 +0000 UTC m=+34.219389326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386672    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:11:00.386655317 +0000 UTC m=+34.219463533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:10:44.386791 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.386694    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:11:00.386683938 +0000 UTC m=+34.219492150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:10:44.741832 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:44.741802    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:44.742308 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:44.741903    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:45.739180 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:45.739149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:45.739385 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:45.739150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:45.739385 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:45.739289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:45.739385 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:45.739335    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:46.742719 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:46.742689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:46.743068 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:46.742794    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:47.738963 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:47.738934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:47.739177 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:47.738974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:47.739177 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:47.739063    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:47.739294 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:47.739167    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:48.739311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:48.739271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:48.739883 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:48.739412    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:49.739303 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:49.739263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:49.739303 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:49.739302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:49.739828 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:49.739388    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:49.739828 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:49.739515    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:50.743484 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.743010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:50.743982 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:50.743618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:50.869643 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.869616    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d1440d8ab02edb51626f8bb4bbcbf86" containerID="806512afe4a216fb2fdfa0f497a3ff26343e0d530f9155e09043c881766207bf" exitCode=0
Jul 23 11:10:50.869776 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.869702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"2d1440d8ab02edb51626f8bb4bbcbf86","Type":"ContainerDied","Data":"806512afe4a216fb2fdfa0f497a3ff26343e0d530f9155e09043c881766207bf"}
Jul 23 11:10:50.871070 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.871051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" event={"ID":"d5ee178cb91c3b2f8f69b5e375b061e5","Type":"ContainerStarted","Data":"e7eb7298f3427e4ef10417b31fec565ec26f6895088315f8b3864a5f53b033d7"}
Jul 23 11:10:50.872401 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.872378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxs7t" event={"ID":"d65b3a24-773e-4ac9-a4eb-615ef57a13aa","Type":"ContainerStarted","Data":"b2a45ec893782f4282b917aa5dda374eb98d25f795ea472d74253d6bb8ce692a"}
Jul 23 11:10:50.873693 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.873669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tvvkn" event={"ID":"e655262a-c02f-4320-b3c9-6b4c22b446b0","Type":"ContainerStarted","Data":"a07f5171fb748c0df3b5fb6bcbced6ce5a585556e7ce3f5e0820f6704f0832c6"}
Jul 23 11:10:50.875013 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.874993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerStarted","Data":"099d20f608b1e50608c6b4a5e0028ff5d705da5769a4aeb7ea684f7ee66e9300"}
Jul 23 11:10:50.877128 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.877084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7nbsm" event={"ID":"498c5f37-ded0-46f2-8784-4598551995eb","Type":"ContainerStarted","Data":"266fd855335043bd1de9fa319f508acf15fa8133fd17bf1e13c37da8c82cfdf2"}
Jul 23 11:10:50.882419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.882393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4jcc8" event={"ID":"dd73de0a-9361-4d9e-9486-a4beb65010c9","Type":"ContainerStarted","Data":"c2dce312bb51438cc338d62e3af0a1d13fa7784195cde792acd3c7309932059b"}
Jul 23 11:10:50.883813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.883794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8" event={"ID":"c51760f5-5a51-4204-b88b-10b587e42069","Type":"ContainerStarted","Data":"edb7796ad09f143abcd4024611a388cb211204ca75965bc5bc5db05731da68b1"}
Jul 23 11:10:50.885932 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.885914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"4daa1f88989b53171dabec1abaf76e812d1c7f8db346c69fdfab56f52d883211"}
Jul 23 11:10:50.886011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.885983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"de7e9039cc6314dadc2d312fedbf49756fafc0957561dbb3ec376dc71b697083"}
Jul 23 11:10:50.886064 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.886024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"3b4c0e2350f80f9bfd4046c6354bb6f7afa0176ce682f033f35d839df6e09433"}
Jul 23 11:10:50.887312 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.887295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shvrq" event={"ID":"1075af97-b1ff-40bc-8742-d35953d18a85","Type":"ContainerStarted","Data":"bc17bc15aa0f61b21bf0bb4a2e116c1aecee4d1629090df25ced5be9b19e8ac6"}
Jul 23 11:10:50.920339 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.920296    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-186.ec2.internal" podStartSLOduration=22.920283533 podStartE2EDuration="22.920283533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:50.920155682 +0000 UTC m=+24.752963939" watchObservedRunningTime="2026-07-23 11:10:50.920283533 +0000 UTC m=+24.753091766"
Jul 23 11:10:50.920831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.920795    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7nbsm" podStartSLOduration=2.41798855 podStartE2EDuration="23.920787358s" totalImagesPullingTime="21.502798808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:50.905073981 +0000 UTC m=+24.737882213" watchObservedRunningTime="2026-07-23 11:10:50.920787358 +0000 UTC m=+24.753595637"
Jul 23 11:10:50.983593 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.983543    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4jcc8" podStartSLOduration=2.406532375 podStartE2EDuration="23.983527113s" totalImagesPullingTime="21.576994738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:50.983313336 +0000 UTC m=+24.816121569" watchObservedRunningTime="2026-07-23 11:10:50.983527113 +0000 UTC m=+24.816335343"
Jul 23 11:10:50.984133 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:50.984074    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hxs7t" podStartSLOduration=2.499896317 podStartE2EDuration="23.984059373s" totalImagesPullingTime="21.484163056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:50.962534228 +0000 UTC m=+24.795342460" watchObservedRunningTime="2026-07-23 11:10:50.984059373 +0000 UTC m=+24.816867605"
Jul 23 11:10:51.001727 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.001662    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tvvkn" podStartSLOduration=2.499379563 podStartE2EDuration="24.001642337s" totalImagesPullingTime="21.502262774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:51.00058025 +0000 UTC m=+24.833388484" watchObservedRunningTime="2026-07-23 11:10:51.001642337 +0000 UTC m=+24.834450572"
Jul 23 11:10:51.015498 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.015451    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-shvrq" podStartSLOduration=10.508480941 podStartE2EDuration="24.015435061s" totalImagesPullingTime="13.50695412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:51.015006702 +0000 UTC m=+24.847814926" watchObservedRunningTime="2026-07-23 11:10:51.015435061 +0000 UTC m=+24.848243294"
Jul 23 11:10:51.739792 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.739610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:51.739987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.739658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:51.739987 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:51.739867    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:51.739987 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:51.739945    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:51.890587 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.890555    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="099d20f608b1e50608c6b4a5e0028ff5d705da5769a4aeb7ea684f7ee66e9300" exitCode=0
Jul 23 11:10:51.891258 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.890645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"099d20f608b1e50608c6b4a5e0028ff5d705da5769a4aeb7ea684f7ee66e9300"}
Jul 23 11:10:51.892125 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.892058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9r5q5" event={"ID":"cc18943f-4684-43f8-b3ec-9fcfd0e7bc26","Type":"ContainerStarted","Data":"8051f46e584cfb0124b7293e711e79bd587e99adb2fef0b138528b9380d29c11"}
Jul 23 11:10:51.895203 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.895178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"87a0415bbd3e669ad51517ec7aa81681600a333bc7e709aa2aa0d6ca4b79c20a"}
Jul 23 11:10:51.895297 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.895257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"d6cc10ff0b61f41a1af3f0dac48f9339086f7758e24d5a1edf4e273e058a3618"}
Jul 23 11:10:51.895340 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.895304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"7ff46cd13ac420c5f402c8c3e811c68b182ec452ee7c391efc29ba7069522c37"}
Jul 23 11:10:51.897059 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.897029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" event={"ID":"2d1440d8ab02edb51626f8bb4bbcbf86","Type":"ContainerStarted","Data":"a83cafda8a99b9e14205ecc725a320ab0d32216a3c7cdda064463b5267501e50"}
Jul 23 11:10:51.928087 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.928034    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-186.ec2.internal" podStartSLOduration=23.928020491 podStartE2EDuration="23.928020491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:51.927956779 +0000 UTC m=+25.760765012" watchObservedRunningTime="2026-07-23 11:10:51.928020491 +0000 UTC m=+25.760828724"
Jul 23 11:10:51.942881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:51.942844    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9r5q5" podStartSLOduration=3.421831857 podStartE2EDuration="24.942821112s" totalImagesPullingTime="21.520989255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:51.942566844 +0000 UTC m=+25.775375077" watchObservedRunningTime="2026-07-23 11:10:51.942821112 +0000 UTC m=+25.775629345"
Jul 23 11:10:52.377133 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.377110    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 11:10:52.608807 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.608599    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:10:52.377127024Z","UUID":"acb54ccf-4839-4a67-b07d-83c4e965caf6","Handler":null,"Name":"","Endpoint":""}
Jul 23 11:10:52.610104 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.610079    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 11:10:52.610179 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.610132    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 11:10:52.739694 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.739660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:52.739889 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:52.739774    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:52.900567 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:52.900491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8" event={"ID":"c51760f5-5a51-4204-b88b-10b587e42069","Type":"ContainerStarted","Data":"207c5107e871d6c74b5c56843ca7fc748567a0c638147f7187d28fc8d7b26ab4"}
Jul 23 11:10:53.146267 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.146233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:53.146872 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.146849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:53.739740 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.739713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:53.739895 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:53.739840    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:53.739895 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.739871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:53.739966 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:53.739937    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:53.905355 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.905263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8" event={"ID":"c51760f5-5a51-4204-b88b-10b587e42069","Type":"ContainerStarted","Data":"891760d793a15a93fe7bbeda460747569ecf4ed188169d5f035ab0fd9bc775d9"}
Jul 23 11:10:53.908919 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.908883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"b60071e666cd2c76f5017d37eefe07bb395a9fdf5390106d81ef3acb281ecb6b"}
Jul 23 11:10:53.909147 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.909122    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:53.909712 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.909691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7nbsm"
Jul 23 11:10:53.928328 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:53.928280    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xh8n8" podStartSLOduration=2.761096161 podStartE2EDuration="26.928262048s" totalImagesPullingTime="24.167165887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:53.927799988 +0000 UTC m=+27.760608220" watchObservedRunningTime="2026-07-23 11:10:53.928262048 +0000 UTC m=+27.761070282"
Jul 23 11:10:54.739575 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:54.739506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:54.739731 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:54.739619    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:55.740045 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:55.739644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:55.740045 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:55.739731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:55.740045 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:55.739947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:55.740045 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:55.740019    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:55.918212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:55.918130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" event={"ID":"9c9061b6-e435-4a9b-92c4-cccb7f5774b2","Type":"ContainerStarted","Data":"d4fb8596c3c97b4b004708fa49ff5a21365936af470db44341331ce9201d4ec1"}
Jul 23 11:10:55.951403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:55.951348    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn" podStartSLOduration=7.387384653 podStartE2EDuration="28.951326375s" totalImagesPullingTime="21.563941722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:10:55.951161077 +0000 UTC m=+29.783969309" watchObservedRunningTime="2026-07-23 11:10:55.951326375 +0000 UTC m=+29.784134609"
Jul 23 11:10:56.740130 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.740077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:56.740808 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:56.740222    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:56.922733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.922537    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:56.922733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.922627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:56.922733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.922673    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:56.943458 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.943357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:56.944085 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:56.944064    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:10:57.739564 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.739396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:57.739692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.739452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:57.739692 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:57.739662    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:57.739938 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:57.739913    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:57.925649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.925529    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="84d14ec2fce40270b4c2c199aa8d9a354edf9c110c7c9654b60daa0255356aaf" exitCode=0
Jul 23 11:10:57.925649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.925611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"84d14ec2fce40270b4c2c199aa8d9a354edf9c110c7c9654b60daa0255356aaf"}
Jul 23 11:10:57.954750 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.954704    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mts9v"]
Jul 23 11:10:57.954931 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.954829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:57.954931 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:57.954915    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:57.955631 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.955578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dv8dx"]
Jul 23 11:10:57.955738 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.955672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:57.955807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:57.955757    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:57.959991 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.959963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm4c2"]
Jul 23 11:10:57.960147 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:57.960076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:57.960210 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:57.960182    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:58.930026 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:58.929993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerStarted","Data":"9f00f777c970c877d55666707b07bb613a373a8f3ec5ffee06441b2adbcc1e35"}
Jul 23 11:10:59.739336 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:59.739306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:10:59.739488 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:59.739342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:10:59.739488 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:59.739410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:10:59.739488 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:59.739428    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:10:59.739575 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:59.739494    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:10:59.739623 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:10:59.739565    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:10:59.934192 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:59.934164    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="9f00f777c970c877d55666707b07bb613a373a8f3ec5ffee06441b2adbcc1e35" exitCode=0
Jul 23 11:10:59.934579 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:10:59.934207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"9f00f777c970c877d55666707b07bb613a373a8f3ec5ffee06441b2adbcc1e35"}
Jul 23 11:11:00.403584 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:00.403541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:00.403755 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:00.403632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:11:00.403755 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:00.403666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:00.403755 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403705    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403753    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403782    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403797    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403806    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wkxd6 for pod openshift-network-diagnostics/network-check-target-mm4c2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403784    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret podName:70cb2736-a420-4e4d-be73-a9a12d0c3f15 nodeName:}" failed. No retries permitted until 2026-07-23 11:11:32.403764755 +0000 UTC m=+66.236572987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret") pod "global-pull-secret-syncer-mts9v" (UID: "70cb2736-a420-4e4d-be73-a9a12d0c3f15") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403854    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:11:32.403842177 +0000 UTC m=+66.236650388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:11:00.403920 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:00.403865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6 podName:d788e41e-1539-4203-9c2a-b95e2a08c822 nodeName:}" failed. No retries permitted until 2026-07-23 11:11:32.403858803 +0000 UTC m=+66.236667013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkxd6" (UniqueName: "kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6") pod "network-check-target-mm4c2" (UID: "d788e41e-1539-4203-9c2a-b95e2a08c822") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:11:00.938317 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:00.938285    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="d381fcfb409101827780cc217933c0e403696805534b22f5a334b61e9f3cb704" exitCode=0
Jul 23 11:11:00.938769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:00.938326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"d381fcfb409101827780cc217933c0e403696805534b22f5a334b61e9f3cb704"}
Jul 23 11:11:01.739292 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:01.739251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:11:01.739292 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:01.739269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:01.739477 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:01.739403    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:11:01.739477 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:01.739454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:01.739568 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:01.739515    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:11:01.739627 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:01.739603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:11:03.739643 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:03.739604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:03.740139 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:03.739648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:03.740139 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:03.739654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:11:03.740139 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:03.739740    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm4c2" podUID="d788e41e-1539-4203-9c2a-b95e2a08c822"
Jul 23 11:11:03.740139 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:03.739822    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mts9v" podUID="70cb2736-a420-4e4d-be73-a9a12d0c3f15"
Jul 23 11:11:03.740139 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:03.739883    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:11:04.513825 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.513792    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-186.ec2.internal" event="NodeReady"
Jul 23 11:11:04.513997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.513975    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 11:11:04.560478 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.560439    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k9ddn"]
Jul 23 11:11:04.592067 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.592027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tqzkh"]
Jul 23 11:11:04.592242 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.592176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.594913 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.594878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 11:11:04.595043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.594998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 11:11:04.595165 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.595138    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hxmt4\""
Jul 23 11:11:04.607117 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.607068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9ddn"]
Jul 23 11:11:04.607268 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.607137    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tqzkh"]
Jul 23 11:11:04.607268 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.607234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.610139 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.610114    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 11:11:04.610280 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.610252    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 11:11:04.610368 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.610354    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-565pp\""
Jul 23 11:11:04.610436 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.610389    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 11:11:04.731624 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4548a142-564b-4bd2-8c29-5218892673ea-config-volume\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.731792 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lsh\" (UniqueName: \"kubernetes.io/projected/fd804822-80f3-4c89-8b6c-ca87eff98f8c-kube-api-access-n7lsh\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.731792 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.731792 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4548a142-564b-4bd2-8c29-5218892673ea-tmp-dir\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.731928 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcp7\" (UniqueName: \"kubernetes.io/projected/4548a142-564b-4bd2-8c29-5218892673ea-kube-api-access-5kcp7\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.731928 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.731856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd804822-80f3-4c89-8b6c-ca87eff98f8c-cert\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.832878 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.832790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd804822-80f3-4c89-8b6c-ca87eff98f8c-cert\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.832878 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.832863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4548a142-564b-4bd2-8c29-5218892673ea-config-volume\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.832893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lsh\" (UniqueName: \"kubernetes.io/projected/fd804822-80f3-4c89-8b6c-ca87eff98f8c-kube-api-access-n7lsh\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.832911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.832992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4548a142-564b-4bd2-8c29-5218892673ea-tmp-dir\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.833060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcp7\" (UniqueName: \"kubernetes.io/projected/4548a142-564b-4bd2-8c29-5218892673ea-kube-api-access-5kcp7\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:04.833105    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:04.833407 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:04.833190    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:05.333148753 +0000 UTC m=+39.165956967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:04.833661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.833425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4548a142-564b-4bd2-8c29-5218892673ea-tmp-dir\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.833718 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.833689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4548a142-564b-4bd2-8c29-5218892673ea-config-volume\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.837550 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.837531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd804822-80f3-4c89-8b6c-ca87eff98f8c-cert\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.843570 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.843547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcp7\" (UniqueName: \"kubernetes.io/projected/4548a142-564b-4bd2-8c29-5218892673ea-kube-api-access-5kcp7\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:04.843859 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.843831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lsh\" (UniqueName: \"kubernetes.io/projected/fd804822-80f3-4c89-8b6c-ca87eff98f8c-kube-api-access-n7lsh\") pod \"ingress-canary-tqzkh\" (UID: \"fd804822-80f3-4c89-8b6c-ca87eff98f8c\") " pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:04.918819 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:04.918784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tqzkh"
Jul 23 11:11:05.337191 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.337150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:05.337382 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:05.337288    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:05.337382 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:05.337377    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:06.33735823 +0000 UTC m=+40.170166442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:05.739529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.739500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:05.739693 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.739610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:11:05.739751 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.739721    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:05.743004 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.742984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 11:11:05.743153 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.743125    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 11:11:05.744452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.744382    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kg9k4\""
Jul 23 11:11:05.744452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.744414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 11:11:05.744676 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.744472    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 11:11:05.744676 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.744523    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twtsh\""
Jul 23 11:11:05.788040 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.788008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tqzkh"]
Jul 23 11:11:05.845639 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:11:05.845595    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd804822_80f3_4c89_8b6c_ca87eff98f8c.slice/crio-50a32848ccf9c49c911806540e36a15c9a9f9017e5437626a226e4ff545ed39c WatchSource:0}: Error finding container 50a32848ccf9c49c911806540e36a15c9a9f9017e5437626a226e4ff545ed39c: Status 404 returned error can't find the container with id 50a32848ccf9c49c911806540e36a15c9a9f9017e5437626a226e4ff545ed39c
Jul 23 11:11:05.951488 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:05.951460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tqzkh" event={"ID":"fd804822-80f3-4c89-8b6c-ca87eff98f8c","Type":"ContainerStarted","Data":"50a32848ccf9c49c911806540e36a15c9a9f9017e5437626a226e4ff545ed39c"}
Jul 23 11:11:06.345825 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:06.345785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:06.345990 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:06.345942    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:06.346030 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:06.346001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:08.345984918 +0000 UTC m=+42.178793135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:06.957892 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:06.957632    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="a09e3f24c79812aa28a9f42afadb51b2cfcc3195b59f358fefd38bd445e61820" exitCode=0
Jul 23 11:11:06.958304 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:06.957708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"a09e3f24c79812aa28a9f42afadb51b2cfcc3195b59f358fefd38bd445e61820"}
Jul 23 11:11:07.964056 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:07.964019    2583 generic.go:356] "Generic (PLEG): container finished" podID="c43c0bf8-f3c5-4b05-8800-c5a7a186703c" containerID="aaeaa996d0d367b3022d092af889c9e617522976335e962867287a4716924b84" exitCode=0
Jul 23 11:11:07.964475 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:07.964111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerDied","Data":"aaeaa996d0d367b3022d092af889c9e617522976335e962867287a4716924b84"}
Jul 23 11:11:08.361518 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:08.361481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:08.361702 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:08.361599    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:08.361702 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:08.361651    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:12.361637596 +0000 UTC m=+46.194445809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:08.970527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:08.970483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" event={"ID":"c43c0bf8-f3c5-4b05-8800-c5a7a186703c","Type":"ContainerStarted","Data":"f99b4d45742dbcdaf1c5f9b86951ca3759fd353b4a18d7042fa0fb72b6473a4e"}
Jul 23 11:11:08.971975 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:08.971952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tqzkh" event={"ID":"fd804822-80f3-4c89-8b6c-ca87eff98f8c","Type":"ContainerStarted","Data":"149c1570e6935b601a56cb9326381d62ba68e8efc060a5688367291e56374198"}
Jul 23 11:11:08.995867 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:08.995797    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5p4hs" podStartSLOduration=8.4070178 podStartE2EDuration="41.995775532s" totalImagesPullingTime="33.588757732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:08.994190254 +0000 UTC m=+42.826998506" watchObservedRunningTime="2026-07-23 11:11:08.995775532 +0000 UTC m=+42.828583765"
Jul 23 11:11:09.012278 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:09.012223    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tqzkh" podStartSLOduration=2.335446917 podStartE2EDuration="5.012201105s" totalImagesPullingTime="2.676754188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:09.011460484 +0000 UTC m=+42.844268717" watchObservedRunningTime="2026-07-23 11:11:09.012201105 +0000 UTC m=+42.845009338"
Jul 23 11:11:12.388671 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:12.388631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:12.389178 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:12.388763    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:12.389178 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:12.388824    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:20.388800041 +0000 UTC m=+54.221608253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:18.746988 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.746955    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"]
Jul 23 11:11:18.784655 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.784618    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"]
Jul 23 11:11:18.784803 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.784743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:18.787558 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.787531    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 11:11:18.787558 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.787560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 11:11:18.787764 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.787570    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 11:11:18.787764 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.787567    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 11:11:18.798400 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.798383    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"]
Jul 23 11:11:18.798478 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.798417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"]
Jul 23 11:11:18.798519 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.798494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.804734 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.804712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 11:11:18.804912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.804747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 11:11:18.804912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.804753    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 11:11:18.804912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.804771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 11:11:18.930765 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.930765 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p289\" (UniqueName: \"kubernetes.io/projected/a4587ecf-fcc2-441b-8ee0-d9602f569807-kube-api-access-5p289\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4587ecf-fcc2-441b-8ee0-d9602f569807-tmp\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59c199a-7790-40ff-b189-f09679ff0e88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.930983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.931011 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.931002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjxw\" (UniqueName: \"kubernetes.io/projected/a59c199a-7790-40ff-b189-f09679ff0e88-kube-api-access-4fjxw\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:18.931250 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:18.931032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a4587ecf-fcc2-441b-8ee0-d9602f569807-klusterlet-config\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.031400 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031400 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4587ecf-fcc2-441b-8ee0-d9602f569807-tmp\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.031400 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59c199a-7790-40ff-b189-f09679ff0e88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjxw\" (UniqueName: \"kubernetes.io/projected/a59c199a-7790-40ff-b189-f09679ff0e88-kube-api-access-4fjxw\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a4587ecf-fcc2-441b-8ee0-d9602f569807-klusterlet-config\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.031737 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031737 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.031834 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.031733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p289\" (UniqueName: \"kubernetes.io/projected/a4587ecf-fcc2-441b-8ee0-d9602f569807-kube-api-access-5p289\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.032177 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.032155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4587ecf-fcc2-441b-8ee0-d9602f569807-tmp\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.032295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.032198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59c199a-7790-40ff-b189-f09679ff0e88-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.035461 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.035429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-ca\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.035585 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.035518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a4587ecf-fcc2-441b-8ee0-d9602f569807-klusterlet-config\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.035585 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.035546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.035666 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.035601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.035699 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.035662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59c199a-7790-40ff-b189-f09679ff0e88-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.041053 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.041028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p289\" (UniqueName: \"kubernetes.io/projected/a4587ecf-fcc2-441b-8ee0-d9602f569807-kube-api-access-5p289\") pod \"klusterlet-addon-workmgr-657556cc79-9vj6n\" (UID: \"a4587ecf-fcc2-441b-8ee0-d9602f569807\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.041307 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.041287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjxw\" (UniqueName: \"kubernetes.io/projected/a59c199a-7790-40ff-b189-f09679ff0e88-kube-api-access-4fjxw\") pod \"cluster-proxy-proxy-agent-6b794b6946-ms5t8\" (UID: \"a59c199a-7790-40ff-b189-f09679ff0e88\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.094259 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.094219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:19.118997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.118961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:11:19.269956 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.269923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"]
Jul 23 11:11:19.273515 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:11:19.273489    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4587ecf_fcc2_441b_8ee0_d9602f569807.slice/crio-f8acf3f067c741d86a78027df17e3a60edf0fec8b98db05bcf9283b2fd01a5fd WatchSource:0}: Error finding container f8acf3f067c741d86a78027df17e3a60edf0fec8b98db05bcf9283b2fd01a5fd: Status 404 returned error can't find the container with id f8acf3f067c741d86a78027df17e3a60edf0fec8b98db05bcf9283b2fd01a5fd
Jul 23 11:11:19.292430 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:19.292360    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"]
Jul 23 11:11:19.295149 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:11:19.295125    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59c199a_7790_40ff_b189_f09679ff0e88.slice/crio-065af9e2c771b8c107a9e02cd1d828835bbb6ed82586e53b64ed59697ec3bc55 WatchSource:0}: Error finding container 065af9e2c771b8c107a9e02cd1d828835bbb6ed82586e53b64ed59697ec3bc55: Status 404 returned error can't find the container with id 065af9e2c771b8c107a9e02cd1d828835bbb6ed82586e53b64ed59697ec3bc55
Jul 23 11:11:20.002159 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:20.002124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerStarted","Data":"065af9e2c771b8c107a9e02cd1d828835bbb6ed82586e53b64ed59697ec3bc55"}
Jul 23 11:11:20.003061 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:20.003031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n" event={"ID":"a4587ecf-fcc2-441b-8ee0-d9602f569807","Type":"ContainerStarted","Data":"f8acf3f067c741d86a78027df17e3a60edf0fec8b98db05bcf9283b2fd01a5fd"}
Jul 23 11:11:20.445010 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:20.444968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:20.445233 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:20.445193    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:20.445303 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:20.445277    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:11:36.445254439 +0000 UTC m=+70.278062653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:26.022300 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.022255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerStarted","Data":"5783a2878c728a906e46a9f4e90c4c1a6c27c7c6c071712ab00f90be4b246292"}
Jul 23 11:11:26.022788 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.022318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerStarted","Data":"9fe62b73a3b4adc7c5b5ae9849f3d6c70f9c212899efb5d72811db6c33e8b634"}
Jul 23 11:11:26.022788 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.022336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerStarted","Data":"6dbc0fdb34b86230d434a246735daadf915fe199b5b89eeed8b2e84729d8cd44"}
Jul 23 11:11:26.023701 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.023676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n" event={"ID":"a4587ecf-fcc2-441b-8ee0-d9602f569807","Type":"ContainerStarted","Data":"53fe6b9dfd3a71ce0c015b827b43992d59ca6059f0f14f26e6269cc4bfe811d5"}
Jul 23 11:11:26.023771 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.023753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:26.025361 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.025336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:11:26.042428 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.042382    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" podStartSLOduration=2.372116771 podStartE2EDuration="8.042368807s" totalImagesPullingTime="5.670252036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:26.041172637 +0000 UTC m=+59.873980898" watchObservedRunningTime="2026-07-23 11:11:26.042368807 +0000 UTC m=+59.875177034"
Jul 23 11:11:26.058293 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:26.058247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n" podStartSLOduration=2.363162026 podStartE2EDuration="8.058230596s" totalImagesPullingTime="5.69506857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:26.057683937 +0000 UTC m=+59.890492190" watchObservedRunningTime="2026-07-23 11:11:26.058230596 +0000 UTC m=+59.891038828"
Jul 23 11:11:28.945737 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:28.945708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hbxgn"
Jul 23 11:11:32.442332 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.442295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:11:32.442332 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.442338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:32.442761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.442370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:32.445262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.445240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 11:11:32.445379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.445323    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 11:11:32.445440 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.445396    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 11:11:32.452910 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:32.452892    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 11:11:32.453007 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:32.452977    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:12:36.452954068 +0000 UTC m=+130.285762283 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : secret "metrics-daemon-secret" not found
Jul 23 11:11:32.455186 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.455168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70cb2736-a420-4e4d-be73-a9a12d0c3f15-original-pull-secret\") pod \"global-pull-secret-syncer-mts9v\" (UID: \"70cb2736-a420-4e4d-be73-a9a12d0c3f15\") " pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:32.455294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.455277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 11:11:32.465728 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.465703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxd6\" (UniqueName: \"kubernetes.io/projected/d788e41e-1539-4203-9c2a-b95e2a08c822-kube-api-access-wkxd6\") pod \"network-check-target-mm4c2\" (UID: \"d788e41e-1539-4203-9c2a-b95e2a08c822\") " pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:32.751913 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.751824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mts9v"
Jul 23 11:11:32.761758 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.761734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-twtsh\""
Jul 23 11:11:32.768431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.768406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:32.888227 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.888195    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mts9v"]
Jul 23 11:11:32.891190 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:11:32.891151    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cb2736_a420_4e4d_be73_a9a12d0c3f15.slice/crio-0b1a478a8b4b2e2d9b622256f26d3253b9bef222ebbf994567bb32a3fd7bdc3f WatchSource:0}: Error finding container 0b1a478a8b4b2e2d9b622256f26d3253b9bef222ebbf994567bb32a3fd7bdc3f: Status 404 returned error can't find the container with id 0b1a478a8b4b2e2d9b622256f26d3253b9bef222ebbf994567bb32a3fd7bdc3f
Jul 23 11:11:32.903088 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:32.903066    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm4c2"]
Jul 23 11:11:32.906153 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:11:32.906124    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd788e41e_1539_4203_9c2a_b95e2a08c822.slice/crio-7b96ae83ae2429c827cd0b53ef7c3b79f12babc78c245de2d470897a89b1b375 WatchSource:0}: Error finding container 7b96ae83ae2429c827cd0b53ef7c3b79f12babc78c245de2d470897a89b1b375: Status 404 returned error can't find the container with id 7b96ae83ae2429c827cd0b53ef7c3b79f12babc78c245de2d470897a89b1b375
Jul 23 11:11:33.042405 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:33.042304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm4c2" event={"ID":"d788e41e-1539-4203-9c2a-b95e2a08c822","Type":"ContainerStarted","Data":"7b96ae83ae2429c827cd0b53ef7c3b79f12babc78c245de2d470897a89b1b375"}
Jul 23 11:11:33.043442 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:33.043418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mts9v" event={"ID":"70cb2736-a420-4e4d-be73-a9a12d0c3f15","Type":"ContainerStarted","Data":"0b1a478a8b4b2e2d9b622256f26d3253b9bef222ebbf994567bb32a3fd7bdc3f"}
Jul 23 11:11:36.471514 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:36.471478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:11:36.471877 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:36.471631    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:11:36.471877 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:11:36.471714    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:12:08.47169753 +0000 UTC m=+102.304505743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:11:38.062073 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:38.062034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm4c2" event={"ID":"d788e41e-1539-4203-9c2a-b95e2a08c822","Type":"ContainerStarted","Data":"73e7e2fa35791aa1cd3713f5412c7e706da470f3f6de92c1bba5ee8e496c475a"}
Jul 23 11:11:38.062524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:38.062149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:11:38.063822 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:38.063797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mts9v" event={"ID":"70cb2736-a420-4e4d-be73-a9a12d0c3f15","Type":"ContainerStarted","Data":"d020c954cf644067c0757fa385b9e9d0a0bd0a30917e01f9ec582641f188d30b"}
Jul 23 11:11:38.080254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:38.080185    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mm4c2" podStartSLOduration=66.084458846 podStartE2EDuration="1m11.08016554s" totalImagesPullingTime="4.995706694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:38.079243446 +0000 UTC m=+71.912051701" watchObservedRunningTime="2026-07-23 11:11:38.08016554 +0000 UTC m=+71.912973778"
Jul 23 11:11:38.094644 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:11:38.094590    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mts9v" podStartSLOduration=65.080626668 podStartE2EDuration="1m10.094570978s" totalImagesPullingTime="5.01394431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:11:38.094318038 +0000 UTC m=+71.927126292" watchObservedRunningTime="2026-07-23 11:11:38.094570978 +0000 UTC m=+71.927379213"
Jul 23 11:12:08.500712 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:12:08.500664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:12:08.501180 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:12:08.500813    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:12:08.501180 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:12:08.500871    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:13:12.500855532 +0000 UTC m=+166.333663743 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:12:09.069795 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:12:09.069765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mm4c2"
Jul 23 11:12:36.493375 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:12:36.493335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:12:36.493771 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:12:36.493452    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 23 11:12:36.493771 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:12:36.493520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs podName:d377a039-1e9c-41f4-a468-52fcc97e49de nodeName:}" failed. No retries permitted until 2026-07-23 11:14:38.493502383 +0000 UTC m=+252.326310595 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs") pod "network-metrics-daemon-dv8dx" (UID: "d377a039-1e9c-41f4-a468-52fcc97e49de") : secret "metrics-daemon-secret" not found
Jul 23 11:13:07.603722 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:13:07.603609    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k9ddn" podUID="4548a142-564b-4bd2-8c29-5218892673ea"
Jul 23 11:13:08.324186 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:08.324154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:13:08.761730 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:13:08.761691    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dv8dx" podUID="d377a039-1e9c-41f4-a468-52fcc97e49de"
Jul 23 11:13:12.543577 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:12.543525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:13:12.543999 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:13:12.543663    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:13:12.543999 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:13:12.543721    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls podName:4548a142-564b-4bd2-8c29-5218892673ea nodeName:}" failed. No retries permitted until 2026-07-23 11:15:14.543707401 +0000 UTC m=+288.376515613 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls") pod "dns-default-k9ddn" (UID: "4548a142-564b-4bd2-8c29-5218892673ea") : secret "dns-default-metrics-tls" not found
Jul 23 11:13:22.739908 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:22.739820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:13:25.373556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:25.373465    2583 generic.go:356] "Generic (PLEG): container finished" podID="a4587ecf-fcc2-441b-8ee0-d9602f569807" containerID="53fe6b9dfd3a71ce0c015b827b43992d59ca6059f0f14f26e6269cc4bfe811d5" exitCode=1
Jul 23 11:13:25.373904 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:25.373546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n" event={"ID":"a4587ecf-fcc2-441b-8ee0-d9602f569807","Type":"ContainerDied","Data":"53fe6b9dfd3a71ce0c015b827b43992d59ca6059f0f14f26e6269cc4bfe811d5"}
Jul 23 11:13:25.373904 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:25.373871    2583 scope.go:117] "RemoveContainer" containerID="53fe6b9dfd3a71ce0c015b827b43992d59ca6059f0f14f26e6269cc4bfe811d5"
Jul 23 11:13:26.024307 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:26.024267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:13:26.381291 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:26.381198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n" event={"ID":"a4587ecf-fcc2-441b-8ee0-d9602f569807","Type":"ContainerStarted","Data":"ed9de36733e1654818b38dae2373adc38b99fc197bcc55d0b179129394f9521f"}
Jul 23 11:13:26.381709 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:26.381391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:13:26.382004 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:13:26.381986    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657556cc79-9vj6n"
Jul 23 11:14:03.888024 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.887989    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmfrz"]
Jul 23 11:14:03.890016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.890000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:03.895766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.895736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 11:14:03.895766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.895736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 11:14:03.896455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.896417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 11:14:03.897529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.897506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r4rm6\""
Jul 23 11:14:03.910629 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.910604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 11:14:03.943789 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.943761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmfrz"]
Jul 23 11:14:03.999254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.999226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:03.999419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.999267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:03.999419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.999289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b021505-da25-4104-a92e-a9ca5348ac58-crio-socket\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:03.999419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.999337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nrb\" (UniqueName: \"kubernetes.io/projected/4b021505-da25-4104-a92e-a9ca5348ac58-kube-api-access-46nrb\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:03.999419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:03.999382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b021505-da25-4104-a92e-a9ca5348ac58-data-volume\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100748 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b021505-da25-4104-a92e-a9ca5348ac58-crio-socket\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nrb\" (UniqueName: \"kubernetes.io/projected/4b021505-da25-4104-a92e-a9ca5348ac58-kube-api-access-46nrb\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b021505-da25-4104-a92e-a9ca5348ac58-data-volume\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.100852    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:14:04.100916 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.100881    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:04.101123 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.100931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4b021505-da25-4104-a92e-a9ca5348ac58-crio-socket\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.101123 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.100951    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:04.600930297 +0000 UTC m=+218.433738529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : configmap "kube-rbac-proxy" not found
Jul 23 11:14:04.101123 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.100980    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:04.6009627 +0000 UTC m=+218.433770914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:04.101235 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.101190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b021505-da25-4104-a92e-a9ca5348ac58-data-volume\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.109876 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.109853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nrb\" (UniqueName: \"kubernetes.io/projected/4b021505-da25-4104-a92e-a9ca5348ac58-kube-api-access-46nrb\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.604389 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.604349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.604389 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:04.604396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:04.604631 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.604496    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:14:04.604631 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.604520    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:04.604631 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.604570    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:05.604556332 +0000 UTC m=+219.437364547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:04.604631 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:04.604585    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:05.60457721 +0000 UTC m=+219.437385425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : configmap "kube-rbac-proxy" not found
Jul 23 11:14:05.614324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:05.614265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:05.614324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:05.614335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:05.614807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:05.614392    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:14:05.614807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:05.614460    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:05.614807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:05.614469    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:07.614453477 +0000 UTC m=+221.447261692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : configmap "kube-rbac-proxy" not found
Jul 23 11:14:05.614807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:05.614516    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:07.614500852 +0000 UTC m=+221.447309066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:05.909910 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:05.909812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvvkn_e655262a-c02f-4320-b3c9-6b4c22b446b0/dns-node-resolver/0.log"
Jul 23 11:14:06.910824 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:06.910792    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shvrq_1075af97-b1ff-40bc-8742-d35953d18a85/node-ca/0.log"
Jul 23 11:14:07.630056 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:07.630001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:07.630056 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:07.630066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:07.630345 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:07.630146    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:14:07.630345 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:07.630212    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:11.630195894 +0000 UTC m=+225.463004105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : configmap "kube-rbac-proxy" not found
Jul 23 11:14:07.630345 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:07.630226    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:07.630345 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:07.630277    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:11.630261936 +0000 UTC m=+225.463070153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:07.911085 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:07.911005    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tqzkh_fd804822-80f3-4c89-8b6c-ca87eff98f8c/serve-healthcheck-canary/0.log"
Jul 23 11:14:11.664434 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:11.664393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:11.664887 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:11.664478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:11.664887 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:11.664547    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:14:11.664887 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:11.664547    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:11.664887 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:11.664610    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:19.664597159 +0000 UTC m=+233.497405373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : configmap "kube-rbac-proxy" not found
Jul 23 11:14:11.664887 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:11.664622    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls podName:4b021505-da25-4104-a92e-a9ca5348ac58 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:19.664616658 +0000 UTC m=+233.497424872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmfrz" (UID: "4b021505-da25-4104-a92e-a9ca5348ac58") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:14:19.722281 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.722237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:19.722674 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.722293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:19.722891 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.722847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4b021505-da25-4104-a92e-a9ca5348ac58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:19.724815 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.724795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4b021505-da25-4104-a92e-a9ca5348ac58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmfrz\" (UID: \"4b021505-da25-4104-a92e-a9ca5348ac58\") " pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:19.799401 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.799355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmfrz"
Jul 23 11:14:19.925347 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:19.925312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmfrz"]
Jul 23 11:14:19.928312 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:14:19.928289    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b021505_da25_4104_a92e_a9ca5348ac58.slice/crio-df94278c005b8ccd18060818a24e3de4133e207d3a394a4c7f8009f2c9d3f1ec WatchSource:0}: Error finding container df94278c005b8ccd18060818a24e3de4133e207d3a394a4c7f8009f2c9d3f1ec: Status 404 returned error can't find the container with id df94278c005b8ccd18060818a24e3de4133e207d3a394a4c7f8009f2c9d3f1ec
Jul 23 11:14:20.535479 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:20.535439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmfrz" event={"ID":"4b021505-da25-4104-a92e-a9ca5348ac58","Type":"ContainerStarted","Data":"df94278c005b8ccd18060818a24e3de4133e207d3a394a4c7f8009f2c9d3f1ec"}
Jul 23 11:14:25.552930 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:25.552891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmfrz" event={"ID":"4b021505-da25-4104-a92e-a9ca5348ac58","Type":"ContainerStarted","Data":"81998f567aaa7e31d9317820d4a69916e2a2216c70dcb8006b56fe94be679be1"}
Jul 23 11:14:29.569627 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:29.569539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmfrz" event={"ID":"4b021505-da25-4104-a92e-a9ca5348ac58","Type":"ContainerStarted","Data":"3a463911cad018a0f1ef983887f6b3d1e629c7bf7e10992a5134f4af1f4af57a"}
Jul 23 11:14:33.588753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:33.588711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmfrz" event={"ID":"4b021505-da25-4104-a92e-a9ca5348ac58","Type":"ContainerStarted","Data":"c8578c308479c36b8e621a79f55e5fd1c36d132b0eacc475909388857d6db8f6"}
Jul 23 11:14:33.607917 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:33.607870    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmfrz" podStartSLOduration=17.955131876 podStartE2EDuration="30.607853648s" totalImagesPullingTime="12.652721772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:14:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:14:33.607282276 +0000 UTC m=+247.440090526" watchObservedRunningTime="2026-07-23 11:14:33.607853648 +0000 UTC m=+247.440661881"
Jul 23 11:14:38.575175 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:38.575120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:14:38.577512 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:38.577489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d377a039-1e9c-41f4-a468-52fcc97e49de-metrics-certs\") pod \"network-metrics-daemon-dv8dx\" (UID: \"d377a039-1e9c-41f4-a468-52fcc97e49de\") " pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:14:38.643757 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:38.643728    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kg9k4\""
Jul 23 11:14:38.651143 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:38.651123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dv8dx"
Jul 23 11:14:38.776934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:38.776907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dv8dx"]
Jul 23 11:14:38.780610 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:14:38.780584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd377a039_1e9c_41f4_a468_52fcc97e49de.slice/crio-7ee649c7d606fa8f1a88279cc6528bf3fb7a6fe979482ace40c07a71846a15e9 WatchSource:0}: Error finding container 7ee649c7d606fa8f1a88279cc6528bf3fb7a6fe979482ace40c07a71846a15e9: Status 404 returned error can't find the container with id 7ee649c7d606fa8f1a88279cc6528bf3fb7a6fe979482ace40c07a71846a15e9
Jul 23 11:14:39.608145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:39.608111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dv8dx" event={"ID":"d377a039-1e9c-41f4-a468-52fcc97e49de","Type":"ContainerStarted","Data":"7ee649c7d606fa8f1a88279cc6528bf3fb7a6fe979482ace40c07a71846a15e9"}
Jul 23 11:14:40.612664 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:40.612625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dv8dx" event={"ID":"d377a039-1e9c-41f4-a468-52fcc97e49de","Type":"ContainerStarted","Data":"0bc94165276e79a1bfd1ca5a3f1182082988fb655f771f7c1bc4e7833fc9db61"}
Jul 23 11:14:40.613168 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:40.612678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dv8dx" event={"ID":"d377a039-1e9c-41f4-a468-52fcc97e49de","Type":"ContainerStarted","Data":"24adbc35434ca1db84a043cd260749fc5c998567e267151e079638083ad98836"}
Jul 23 11:14:40.630640 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:40.630591    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dv8dx" podStartSLOduration=251.697439297 podStartE2EDuration="4m12.630577518s" totalImagesPullingTime="933.138221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:14:40.630012472 +0000 UTC m=+254.462820705" watchObservedRunningTime="2026-07-23 11:14:40.630577518 +0000 UTC m=+254.463385750"
Jul 23 11:14:43.252048 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.252014    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:14:43.254642 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.254626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.257692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.257669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 23 11:14:43.257835 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.257695    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 23 11:14:43.258882 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.258856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 11:14:43.259079 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.259039    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqhsh\""
Jul 23 11:14:43.259208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.259088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 23 11:14:43.259208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.259116    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 11:14:43.259208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.259053    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 23 11:14:43.259208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.259162    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 23 11:14:43.264338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.264314    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 23 11:14:43.264770 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.264749    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:14:43.309387 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309387 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2lw\" (UniqueName: \"kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.309713 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.309577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410236 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2lw\" (UniqueName: \"kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.410993 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.410968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.411141 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.411114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.411219 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.411148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.411353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.411334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.413264 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.413245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.413469 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.413448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.419116 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.419076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2lw\" (UniqueName: \"kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw\") pod \"console-5b546b5b9c-f4b9l\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") " pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.565016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.564919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:43.697445 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:43.697414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:14:43.700312 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:14:43.700287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb4fc42_e3e0_4148_b415_2462a2f4384e.slice/crio-bd9ab25d50ec2122f59d77123d3de5a2145ab567299cc5fab1a09ac835ea6172 WatchSource:0}: Error finding container bd9ab25d50ec2122f59d77123d3de5a2145ab567299cc5fab1a09ac835ea6172: Status 404 returned error can't find the container with id bd9ab25d50ec2122f59d77123d3de5a2145ab567299cc5fab1a09ac835ea6172
Jul 23 11:14:44.625614 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:44.625575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b546b5b9c-f4b9l" event={"ID":"3fb4fc42-e3e0-4148-b415-2462a2f4384e","Type":"ContainerStarted","Data":"bd9ab25d50ec2122f59d77123d3de5a2145ab567299cc5fab1a09ac835ea6172"}
Jul 23 11:14:46.635394 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:46.635303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b546b5b9c-f4b9l" event={"ID":"3fb4fc42-e3e0-4148-b415-2462a2f4384e","Type":"ContainerStarted","Data":"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"}
Jul 23 11:14:46.655164 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:46.655108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b546b5b9c-f4b9l" podStartSLOduration=0.974643563 podStartE2EDuration="3.655078706s" totalImagesPullingTime="2.680435143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:14:46.653559092 +0000 UTC m=+260.486367324" watchObservedRunningTime="2026-07-23 11:14:46.655078706 +0000 UTC m=+260.487886939"
Jul 23 11:14:52.396038 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.395956    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-792pm"]
Jul 23 11:14:52.457335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.457307    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.460573 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.460548    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 11:14:52.461076 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.461060    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 11:14:52.461912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.461893    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 11:14:52.462001 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.461936    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 11:14:52.462058 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.462040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bfcwj\""
Jul 23 11:14:52.462187 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.462173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 11:14:52.462885 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.462862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 11:14:52.464330 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-sys\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464404 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-root\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464404 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-textfile\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464495 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464495 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv4f\" (UniqueName: \"kubernetes.io/projected/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-kube-api-access-twv4f\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-wtmp\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.464673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.464618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-metrics-client-ca\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565336 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565336 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-metrics-client-ca\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-sys\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-root\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-textfile\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twv4f\" (UniqueName: \"kubernetes.io/projected/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-kube-api-access-twv4f\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-root\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-wtmp\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-sys\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565909 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-wtmp\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.565909 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:52.565628    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 23 11:14:52.565909 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:14:52.565700    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls podName:f3e17a55-6ab8-4f46-a232-56b3d5ef7c10 nodeName:}" failed. No retries permitted until 2026-07-23 11:14:53.065680468 +0000 UTC m=+266.898488680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls") pod "node-exporter-792pm" (UID: "f3e17a55-6ab8-4f46-a232-56b3d5ef7c10") : secret "node-exporter-tls" not found
Jul 23 11:14:52.566012 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-accelerators-collector-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.566012 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.565966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-metrics-client-ca\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.566087 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.566017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-textfile\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.567999 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.567981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:52.581659 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:52.581636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv4f\" (UniqueName: \"kubernetes.io/projected/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-kube-api-access-twv4f\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:53.068014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.067975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:53.070354 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.070334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3e17a55-6ab8-4f46-a232-56b3d5ef7c10-node-exporter-tls\") pod \"node-exporter-792pm\" (UID: \"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10\") " pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:53.367318 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.367234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-792pm"
Jul 23 11:14:53.424386 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:14:53.424357    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e17a55_6ab8_4f46_a232_56b3d5ef7c10.slice/crio-8f84587eb36008521dddcf7efbf6f5332a9b25f741ab7b08ca789f7a53a9fae7 WatchSource:0}: Error finding container 8f84587eb36008521dddcf7efbf6f5332a9b25f741ab7b08ca789f7a53a9fae7: Status 404 returned error can't find the container with id 8f84587eb36008521dddcf7efbf6f5332a9b25f741ab7b08ca789f7a53a9fae7
Jul 23 11:14:53.565733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.565697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:53.565905 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.565761    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:53.570425 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.570404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:53.659927 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.659847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-792pm" event={"ID":"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10","Type":"ContainerStarted","Data":"8f84587eb36008521dddcf7efbf6f5332a9b25f741ab7b08ca789f7a53a9fae7"}
Jul 23 11:14:53.663556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:53.663536    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:14:55.668057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:55.668020    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3e17a55-6ab8-4f46-a232-56b3d5ef7c10" containerID="68b239fde1407cab13f4342b720441d6981c24b9df3b0e6d4e992abe06df21c9" exitCode=0
Jul 23 11:14:55.668519 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:55.668087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-792pm" event={"ID":"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10","Type":"ContainerDied","Data":"68b239fde1407cab13f4342b720441d6981c24b9df3b0e6d4e992abe06df21c9"}
Jul 23 11:14:56.673684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:56.673646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-792pm" event={"ID":"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10","Type":"ContainerStarted","Data":"009e556955bf7783b8c7bf8521b11e4af3068282dc5f836fc0ad2d593bd29fe2"}
Jul 23 11:14:56.674138 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:56.673704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-792pm" event={"ID":"f3e17a55-6ab8-4f46-a232-56b3d5ef7c10","Type":"ContainerStarted","Data":"bc98fa6d6d644c7a0d707baefa13065e6d025c780890648105f79200ca5d79de"}
Jul 23 11:14:56.696617 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:56.696569    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-792pm" podStartSLOduration=3.148593924 podStartE2EDuration="4.69655313s" totalImagesPullingTime="1.475520637s" totalInitContainerRuntime="72.438569ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:14:56.694630543 +0000 UTC m=+270.527438799" watchObservedRunningTime="2026-07-23 11:14:56.69655313 +0000 UTC m=+270.529361747"
Jul 23 11:14:58.930598 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.930563    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:14:58.936476 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.936452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:58.939505 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.939480    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4kzqk\""
Jul 23 11:14:58.939893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.939857    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 11:14:58.939893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.939867    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-12p6pdg5a3ad\""
Jul 23 11:14:58.940164 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940149    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 11:14:58.940783 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 11:14:58.940783 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940656    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 11:14:58.940783 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940666    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 11:14:58.940783 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 11:14:58.941052 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.940865    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 11:14:58.941324 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.941306    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 11:14:58.942189 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.942170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 11:14:58.942295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.942216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 11:14:58.942462 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.942445    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 11:14:58.942871 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.942844    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 11:14:58.943416 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.943397    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 11:14:58.944796 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.944778    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 11:14:58.948705 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.948685    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 11:14:58.954554 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:58.954530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:14:59.012587 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012759 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012759 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012759 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012865 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012865 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012865 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlp87\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-kube-api-access-jlp87\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012865 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.012985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.012993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.013010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.013058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.013112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.013148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.013197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.013172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.113813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.113776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.113813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.113812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114032 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.113833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114032 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.113956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114032 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.113993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114032 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlp87\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-kube-api-access-jlp87\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.114692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.115017 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.114944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.115157 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.115123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.115620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.115595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.117506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.117481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.118399 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.118819 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.118819 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.118819 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.119026 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.119026 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.118914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.119598 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.119333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.119598 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.119521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.119598 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.119521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.120159 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.120083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.120159 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.120148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d955b53-25f7-4383-959a-0c1b36108937-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.120326 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.120288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.120690 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.120669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.120875 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.120855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.121492 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.121468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d955b53-25f7-4383-959a-0c1b36108937-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.150649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.150618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlp87\" (UniqueName: \"kubernetes.io/projected/3d955b53-25f7-4383-959a-0c1b36108937-kube-api-access-jlp87\") pod \"prometheus-k8s-0\" (UID: \"3d955b53-25f7-4383-959a-0c1b36108937\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.247295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.247207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:14:59.441084 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.441055    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:14:59.443801 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:14:59.443773    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d955b53_25f7_4383_959a_0c1b36108937.slice/crio-d2857d88a411686415ac799f447b999d43a8152f8e00d8905f155866e22f7c0e WatchSource:0}: Error finding container d2857d88a411686415ac799f447b999d43a8152f8e00d8905f155866e22f7c0e: Status 404 returned error can't find the container with id d2857d88a411686415ac799f447b999d43a8152f8e00d8905f155866e22f7c0e
Jul 23 11:14:59.685323 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:14:59.685287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"d2857d88a411686415ac799f447b999d43a8152f8e00d8905f155866e22f7c0e"}
Jul 23 11:15:01.693955 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:01.693918    2583 generic.go:356] "Generic (PLEG): container finished" podID="3d955b53-25f7-4383-959a-0c1b36108937" containerID="b6d09df3b10f2efab3682e4e192c0ff1fcf975f05a9ce35d22027745888c1d22" exitCode=0
Jul 23 11:15:01.694460 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:01.693972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerDied","Data":"b6d09df3b10f2efab3682e4e192c0ff1fcf975f05a9ce35d22027745888c1d22"}
Jul 23 11:15:05.713054 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:05.713008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"8e38af78ff810c52b2c188d4c12c23bb69006c1eb9afe42be6fd88864e701001"}
Jul 23 11:15:05.713622 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:05.713122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"6bcfcf49bb29d9e78995cdff5f9ba577ecd5ceb477fae20ebd34a720325ddc3c"}
Jul 23 11:15:08.726666 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:08.726629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"ec7b50fc854f41fb8b27043303d80ad2ffd47353a3a7cd0152cdc6c578dcf821"}
Jul 23 11:15:08.727193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:08.726744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"dbfccabda91c90da3e1e9457bfbc25a35277ba42cbff88a35fa4cb158e2e9aac"}
Jul 23 11:15:08.727193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:08.726811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"f6971b2bc9c0a586ec52afdfba33e65492f83de162f61e39f09e9b51708bba13"}
Jul 23 11:15:08.727193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:08.726850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d955b53-25f7-4383-959a-0c1b36108937","Type":"ContainerStarted","Data":"eb22c28075234cf34be665282eed71044a6be4cf1ccc0c2061ef5b7c7c121a4f"}
Jul 23 11:15:08.761315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:08.761257    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.94358475 podStartE2EDuration="10.761240203s" totalImagesPullingTime="7.794022652s" totalInitContainerRuntime="23.632801ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:14:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:15:08.759173003 +0000 UTC m=+282.591981261" watchObservedRunningTime="2026-07-23 11:15:08.761240203 +0000 UTC m=+282.594048436"
Jul 23 11:15:09.248376 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:09.248339    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:15:11.325046 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:15:11.324998    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k9ddn" podUID="4548a142-564b-4bd2-8c29-5218892673ea"
Jul 23 11:15:11.738342 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:11.738312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:14.546684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:14.546646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:14.549213 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:14.549190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4548a142-564b-4bd2-8c29-5218892673ea-metrics-tls\") pod \"dns-default-k9ddn\" (UID: \"4548a142-564b-4bd2-8c29-5218892673ea\") " pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:14.742355 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:14.742328    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hxmt4\""
Jul 23 11:15:14.750514 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:14.750491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:14.889274 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:14.889251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9ddn"]
Jul 23 11:15:14.891270 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:15:14.891246    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4548a142_564b_4bd2_8c29_5218892673ea.slice/crio-dc9252696de7f4797432719bd3b09f0896a4e05f82341e02ea7f7ac6e4a7ae7b WatchSource:0}: Error finding container dc9252696de7f4797432719bd3b09f0896a4e05f82341e02ea7f7ac6e4a7ae7b: Status 404 returned error can't find the container with id dc9252696de7f4797432719bd3b09f0896a4e05f82341e02ea7f7ac6e4a7ae7b
Jul 23 11:15:15.357032 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:15.356994    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:15:15.753016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:15.752976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9ddn" event={"ID":"4548a142-564b-4bd2-8c29-5218892673ea","Type":"ContainerStarted","Data":"dc9252696de7f4797432719bd3b09f0896a4e05f82341e02ea7f7ac6e4a7ae7b"}
Jul 23 11:15:17.762583 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:17.762545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9ddn" event={"ID":"4548a142-564b-4bd2-8c29-5218892673ea","Type":"ContainerStarted","Data":"f2698809e96b46dc014d81ec040f19e67120dc0b159ce03f654f506a1fd1be43"}
Jul 23 11:15:17.763004 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:17.762632    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9ddn" event={"ID":"4548a142-564b-4bd2-8c29-5218892673ea","Type":"ContainerStarted","Data":"3efd13eba211a16ec1c043f8203ac465b7efac00be0d7a7124163a0c2bd520d6"}
Jul 23 11:15:17.763004 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:17.762697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:19.120829 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:19.120782    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" podUID="a59c199a-7790-40ff-b189-f09679ff0e88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 11:15:26.557654 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:26.557626    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 11:15:27.768685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:27.768653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k9ddn"
Jul 23 11:15:27.787852 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:27.787802    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k9ddn" podStartSLOduration=261.88444173 podStartE2EDuration="4m23.787785555s" totalImagesPullingTime="1.903343825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:15:17.797183819 +0000 UTC m=+291.629992065" watchObservedRunningTime="2026-07-23 11:15:27.787785555 +0000 UTC m=+301.620593787"
Jul 23 11:15:29.120686 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:29.120648    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" podUID="a59c199a-7790-40ff-b189-f09679ff0e88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 11:15:39.120474 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.120429    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" podUID="a59c199a-7790-40ff-b189-f09679ff0e88" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 23 11:15:39.120931 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.120516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8"
Jul 23 11:15:39.120999 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.120972    2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"5783a2878c728a906e46a9f4e90c4c1a6c27c7c6c071712ab00f90be4b246292"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 23 11:15:39.121037 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.121020    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" podUID="a59c199a-7790-40ff-b189-f09679ff0e88" containerName="service-proxy" containerID="cri-o://5783a2878c728a906e46a9f4e90c4c1a6c27c7c6c071712ab00f90be4b246292" gracePeriod=30
Jul 23 11:15:39.838579 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.838546    2583 generic.go:356] "Generic (PLEG): container finished" podID="a59c199a-7790-40ff-b189-f09679ff0e88" containerID="5783a2878c728a906e46a9f4e90c4c1a6c27c7c6c071712ab00f90be4b246292" exitCode=2
Jul 23 11:15:39.838742 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.838622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerDied","Data":"5783a2878c728a906e46a9f4e90c4c1a6c27c7c6c071712ab00f90be4b246292"}
Jul 23 11:15:39.838742 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:39.838666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b794b6946-ms5t8" event={"ID":"a59c199a-7790-40ff-b189-f09679ff0e88","Type":"ContainerStarted","Data":"98f4747d6689c75634105ac03c018533cd1a9b87cd52dc52e531cf266ad8079f"}
Jul 23 11:15:40.382063 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.382023    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b546b5b9c-f4b9l" podUID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" containerName="console" containerID="cri-o://37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3" gracePeriod=15
Jul 23 11:15:40.612779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.612755    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:15:40.756986 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.756956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757005    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b2lw\" (UniqueName: \"kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757143    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca\") pod \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\" (UID: \"3fb4fc42-e3e0-4148-b415-2462a2f4384e\") "
Jul 23 11:15:40.757559 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757509    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:15:40.757698 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:15:40.757698 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca" (OuterVolumeSpecName: "service-ca") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:15:40.757698 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.757675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config" (OuterVolumeSpecName: "console-config") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:15:40.759522 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.759496    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw" (OuterVolumeSpecName: "kube-api-access-4b2lw") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "kube-api-access-4b2lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:15:40.759522 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.759518    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:15:40.759654 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.759540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3fb4fc42-e3e0-4148-b415-2462a2f4384e" (UID: "3fb4fc42-e3e0-4148-b415-2462a2f4384e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:15:40.843031 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.842994    2583 generic.go:356] "Generic (PLEG): container finished" podID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" containerID="37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3" exitCode=0
Jul 23 11:15:40.843197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.843037    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b546b5b9c-f4b9l"
Jul 23 11:15:40.843197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.843054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b546b5b9c-f4b9l" event={"ID":"3fb4fc42-e3e0-4148-b415-2462a2f4384e","Type":"ContainerDied","Data":"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"}
Jul 23 11:15:40.843197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.843132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b546b5b9c-f4b9l" event={"ID":"3fb4fc42-e3e0-4148-b415-2462a2f4384e","Type":"ContainerDied","Data":"bd9ab25d50ec2122f59d77123d3de5a2145ab567299cc5fab1a09ac835ea6172"}
Jul 23 11:15:40.843197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.843156    2583 scope.go:117] "RemoveContainer" containerID="37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"
Jul 23 11:15:40.851715 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.851696    2583 scope.go:117] "RemoveContainer" containerID="37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"
Jul 23 11:15:40.851985 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:15:40.851962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3\": container with ID starting with 37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3 not found: ID does not exist" containerID="37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"
Jul 23 11:15:40.852082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.851989    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3"} err="failed to get container status \"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3\": rpc error: code = NotFound desc = could not find container \"37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3\": container with ID starting with 37cf3a1ce36aa9c837f22a3f35b094c21daa4174bcc5b794e68d4fbbd97c89a3 not found: ID does not exist"
Jul 23 11:15:40.858313 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858295    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-trusted-ca-bundle\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858316    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-config\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858325    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-serving-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858333    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b2lw\" (UniqueName: \"kubernetes.io/projected/3fb4fc42-e3e0-4148-b415-2462a2f4384e-kube-api-access-4b2lw\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858343    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fb4fc42-e3e0-4148-b415-2462a2f4384e-console-oauth-config\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858351    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-oauth-serving-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.858372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.858358    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb4fc42-e3e0-4148-b415-2462a2f4384e-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:15:40.863804 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.863783    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:15:40.867383 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:40.867363    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b546b5b9c-f4b9l"]
Jul 23 11:15:42.743758 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:42.743721    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" path="/var/lib/kubelet/pods/3fb4fc42-e3e0-4148-b415-2462a2f4384e/volumes"
Jul 23 11:15:59.248349 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:59.248314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:15:59.268731 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:59.268707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:15:59.934331 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:15:59.934297    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:18:38.294884 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.294847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"]
Jul 23 11:18:38.295352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.295084    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" containerName="console"
Jul 23 11:18:38.295352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.295163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" containerName="console"
Jul 23 11:18:38.295352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.295214    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb4fc42-e3e0-4148-b415-2462a2f4384e" containerName="console"
Jul 23 11:18:38.296712 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.296697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.299602 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.299578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:18:38.299602 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.299591    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 11:18:38.299769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.299652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 11:18:38.300722 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.300706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 11:18:38.300797 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.300706    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mdr24\""
Jul 23 11:18:38.300797 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.300749    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 11:18:38.331160 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.331130    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"]
Jul 23 11:18:38.382938 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.382902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.383173 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.382957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqsz8\" (UniqueName: \"kubernetes.io/projected/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-kube-api-access-qqsz8\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.383173 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.383046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-metrics-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.383173 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.383084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-manager-config\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.483852 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.483826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.484027 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.483876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqsz8\" (UniqueName: \"kubernetes.io/projected/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-kube-api-access-qqsz8\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.484027 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.483903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-metrics-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.484027 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.483965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-manager-config\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.484638 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.484615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-manager-config\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.486373 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.486348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-metrics-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.486465 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.486455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-cert\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.494572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.494546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqsz8\" (UniqueName: \"kubernetes.io/projected/12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890-kube-api-access-qqsz8\") pod \"lws-controller-manager-57b4b866cd-pzmkk\" (UID: \"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890\") " pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.525965 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.525934    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"]
Jul 23 11:18:38.528072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.528055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.531397 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.531375    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 11:18:38.532152 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.532124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 11:18:38.532254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.532160    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 11:18:38.532254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.532176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 11:18:38.532254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.532176    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6vh5\""
Jul 23 11:18:38.545543 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.545489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"]
Jul 23 11:18:38.584538 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.584505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-manifests\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.584538 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.584538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-tmp\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.584779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.584561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6m8b\" (UniqueName: \"kubernetes.io/projected/06300105-ae5b-47a7-a6c6-1b99373abb49-kube-api-access-g6m8b\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.584779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.584583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.584779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.584679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-webhook-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.605546 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.605519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:38.686193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.685874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.686193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.685919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-webhook-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.686193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.685969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-manifests\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.686193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.685990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-tmp\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.686193 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.686008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6m8b\" (UniqueName: \"kubernetes.io/projected/06300105-ae5b-47a7-a6c6-1b99373abb49-kube-api-access-g6m8b\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.686547 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.686476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-manifests\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.688712 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.688667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-webhook-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.689285 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.689260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/06300105-ae5b-47a7-a6c6-1b99373abb49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.690333 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.690308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06300105-ae5b-47a7-a6c6-1b99373abb49-tmp\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.696229 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.696207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6m8b\" (UniqueName: \"kubernetes.io/projected/06300105-ae5b-47a7-a6c6-1b99373abb49-kube-api-access-g6m8b\") pod \"opendatahub-operator-controller-manager-684564f56d-p9rdw\" (UID: \"06300105-ae5b-47a7-a6c6-1b99373abb49\") " pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.749072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.749045    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"]
Jul 23 11:18:38.751605 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:18:38.751580    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12aadc1e_4ae6_4639_b9ed_ff4e5dfe5890.slice/crio-3c57217bbf8a5a1dc10327be1a6da0fe475f0858dd1cc9d5b3e73a37b08620be WatchSource:0}: Error finding container 3c57217bbf8a5a1dc10327be1a6da0fe475f0858dd1cc9d5b3e73a37b08620be: Status 404 returned error can't find the container with id 3c57217bbf8a5a1dc10327be1a6da0fe475f0858dd1cc9d5b3e73a37b08620be
Jul 23 11:18:38.753348 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.753331    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:18:38.838628 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.838539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:38.971966 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:38.971939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"]
Jul 23 11:18:38.975770 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:18:38.975743    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06300105_ae5b_47a7_a6c6_1b99373abb49.slice/crio-072efac3eea60e4d21ed006f28313370c48e110516f14f9399fd1dd7802c894f WatchSource:0}: Error finding container 072efac3eea60e4d21ed006f28313370c48e110516f14f9399fd1dd7802c894f: Status 404 returned error can't find the container with id 072efac3eea60e4d21ed006f28313370c48e110516f14f9399fd1dd7802c894f
Jul 23 11:18:39.475484 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:39.475444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw" event={"ID":"06300105-ae5b-47a7-a6c6-1b99373abb49","Type":"ContainerStarted","Data":"072efac3eea60e4d21ed006f28313370c48e110516f14f9399fd1dd7802c894f"}
Jul 23 11:18:39.476811 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:39.476781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk" event={"ID":"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890","Type":"ContainerStarted","Data":"3c57217bbf8a5a1dc10327be1a6da0fe475f0858dd1cc9d5b3e73a37b08620be"}
Jul 23 11:18:43.494162 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:43.494124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw" event={"ID":"06300105-ae5b-47a7-a6c6-1b99373abb49","Type":"ContainerStarted","Data":"544bdddc0205537ce6c4381b79bb55e3a49a45774a2124cbf7b2015089bc24c7"}
Jul 23 11:18:43.495601 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:43.495575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk" event={"ID":"12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890","Type":"ContainerStarted","Data":"f771665dc897557b47a7e7ed93b623c26f91329fa90f1e8657f01f3422ab9251"}
Jul 23 11:18:43.495732 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:43.495712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:43.537305 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:43.537253    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk" podStartSLOduration=1.256655448 podStartE2EDuration="5.537236067s" totalImagesPullingTime="4.280580619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:18:43.536049158 +0000 UTC m=+497.368857390" watchObservedRunningTime="2026-07-23 11:18:43.537236067 +0000 UTC m=+497.370044303"
Jul 23 11:18:44.500556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:44.500474    2583 generic.go:356] "Generic (PLEG): container finished" podID="06300105-ae5b-47a7-a6c6-1b99373abb49" containerID="544bdddc0205537ce6c4381b79bb55e3a49a45774a2124cbf7b2015089bc24c7" exitCode=0
Jul 23 11:18:44.500973 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:44.500570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw" event={"ID":"06300105-ae5b-47a7-a6c6-1b99373abb49","Type":"ContainerDied","Data":"544bdddc0205537ce6c4381b79bb55e3a49a45774a2124cbf7b2015089bc24c7"}
Jul 23 11:18:45.505973 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:45.505937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw" event={"ID":"06300105-ae5b-47a7-a6c6-1b99373abb49","Type":"ContainerStarted","Data":"a46742bb024802a3114c91d1851fc5b884d38cf71810ff0f0112f8bdaa682ef6"}
Jul 23 11:18:45.506476 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:45.506049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:18:45.532725 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:45.532677    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw" podStartSLOduration=1.939347224 podStartE2EDuration="7.532661727s" totalImagesPullingTime="4.496680533s" totalInitContainerRuntime="1.09663397s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:18:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:18:45.530538067 +0000 UTC m=+499.363346312" watchObservedRunningTime="2026-07-23 11:18:45.532661727 +0000 UTC m=+499.365469960"
Jul 23 11:18:54.502683 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:54.502650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57b4b866cd-pzmkk"
Jul 23 11:18:56.511716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:18:56.511686    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-684564f56d-p9rdw"
Jul 23 11:19:06.464120 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.464073    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"]
Jul 23 11:19:06.474347 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.474319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.477628 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.477607    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5ghbw\""
Jul 23 11:19:06.477746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.477609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 11:19:06.477813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.477791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 11:19:06.477875 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.477842    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 11:19:06.489083 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.489063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"]
Jul 23 11:19:06.610490 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610490 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610841 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27r6c\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-kube-api-access-27r6c\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.610841 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.610739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711127 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27r6c\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-kube-api-access-27r6c\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711311 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711657 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711657 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711657 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711853 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.711914 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.711889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.712158 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.712084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.712158 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.712110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.712766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.712732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.714040 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.714018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.714190 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.714119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.722986 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.722966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27r6c\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-kube-api-access-27r6c\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.725149 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.725125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d778b67-9a4c-4b03-8e5d-ad26da471f71-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vnwg6\" (UID: \"6d778b67-9a4c-4b03-8e5d-ad26da471f71\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.783599 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.783569    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:06.981081 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:06.981048    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"]
Jul 23 11:19:06.981485 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:19:06.981464    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d778b67_9a4c_4b03_8e5d_ad26da471f71.slice/crio-59e67ce574c6021a2d1e5757e6de0c24af41585a6594a4af721de25660d8941a WatchSource:0}: Error finding container 59e67ce574c6021a2d1e5757e6de0c24af41585a6594a4af721de25660d8941a: Status 404 returned error can't find the container with id 59e67ce574c6021a2d1e5757e6de0c24af41585a6594a4af721de25660d8941a
Jul 23 11:19:07.592596 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:07.592554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6" event={"ID":"6d778b67-9a4c-4b03-8e5d-ad26da471f71","Type":"ContainerStarted","Data":"59e67ce574c6021a2d1e5757e6de0c24af41585a6594a4af721de25660d8941a"}
Jul 23 11:19:09.150518 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.150467    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"]
Jul 23 11:19:09.155434 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.155408    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.159057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.159030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 11:19:09.159208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.159036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dh9p7\""
Jul 23 11:19:09.159208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.159172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 11:19:09.159373 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.159355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 11:19:09.176379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.176350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"]
Jul 23 11:19:09.233751 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082d1fd-0256-4f1f-8a97-01580c15e3aa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.233940 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.233940 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.233940 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxqg\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-kube-api-access-kjxqg\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.233940 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.234148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.233966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.234148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.234017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.334795 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.334795 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxqg\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-kube-api-access-kjxqg\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.334963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335043 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.335026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082d1fd-0256-4f1f-8a97-01580c15e3aa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.335823 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.335775    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.337723 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.337700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082d1fd-0256-4f1f-8a97-01580c15e3aa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.337893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.337875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.337996 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.337975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.338134 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.338062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9082d1fd-0256-4f1f-8a97-01580c15e3aa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.344624 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.344592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.344728 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.344700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxqg\" (UniqueName: \"kubernetes.io/projected/9082d1fd-0256-4f1f-8a97-01580c15e3aa-kube-api-access-kjxqg\") pod \"istiod-openshift-gateway-94bb8fbfd-wjfns\" (UID: \"9082d1fd-0256-4f1f-8a97-01580c15e3aa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.467123 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.467074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:09.795105 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.795053    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:19:09.795209 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.795152    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:19:09.795209 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.795184    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:19:09.900463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:09.900430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"]
Jul 23 11:19:09.903664 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:19:09.903629    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9082d1fd_0256_4f1f_8a97_01580c15e3aa.slice/crio-88f13c15113804ab5c87eeb90b87fe443a80abefa2b4f204f75711c110c09318 WatchSource:0}: Error finding container 88f13c15113804ab5c87eeb90b87fe443a80abefa2b4f204f75711c110c09318: Status 404 returned error can't find the container with id 88f13c15113804ab5c87eeb90b87fe443a80abefa2b4f204f75711c110c09318
Jul 23 11:19:10.606195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:10.606019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6" event={"ID":"6d778b67-9a4c-4b03-8e5d-ad26da471f71","Type":"ContainerStarted","Data":"fbee3cc51632356dd3c8ac076f2f75d7b4a93c26b341a41bcaf2dfee02675fa1"}
Jul 23 11:19:10.607295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:10.607271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns" event={"ID":"9082d1fd-0256-4f1f-8a97-01580c15e3aa","Type":"ContainerStarted","Data":"88f13c15113804ab5c87eeb90b87fe443a80abefa2b4f204f75711c110c09318"}
Jul 23 11:19:10.656581 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:10.656523    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6" podStartSLOduration=1.845195827 podStartE2EDuration="4.656502222s" totalImagesPullingTime="2.811306395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:19:10.655486382 +0000 UTC m=+524.488294616" watchObservedRunningTime="2026-07-23 11:19:10.656502222 +0000 UTC m=+524.489310455"
Jul 23 11:19:10.783939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:10.783905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:10.789424 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:10.789395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:11.611397 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:11.611367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:11.612720 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:11.612690    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vnwg6"
Jul 23 11:19:12.379103 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:12.379066    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:19:12.379175 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:12.379147    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:19:12.616200 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:12.616148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns" event={"ID":"9082d1fd-0256-4f1f-8a97-01580c15e3aa","Type":"ContainerStarted","Data":"b4f3ac0acfc648583b4cd293dd8760f14f8f1766d36f07c31633e05fc5a9ac1d"}
Jul 23 11:19:12.616592 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:12.616308    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:13.621544 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:13.621464    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns"
Jul 23 11:19:13.648780 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:19:13.648731    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wjfns" podStartSLOduration=2.176079237 podStartE2EDuration="4.648714652s" totalImagesPullingTime="2.472635415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:19:12.67534201 +0000 UTC m=+526.508150244" watchObservedRunningTime="2026-07-23 11:19:13.648714652 +0000 UTC m=+527.481522884"
Jul 23 11:21:38.481830 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.481794    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:21:38.483939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.483920    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.486753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.486728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 11:21:38.486840 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.486756    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 11:21:38.487943 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.487926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-srkvm\""
Jul 23 11:21:38.488019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.487926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 11:21:38.494447 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.494424    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:21:38.531494 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.531459    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:21:38.565260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.565224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/01191d10-8c6a-4454-a048-ebc1b9cd3593-config-file\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.565394 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.565266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6dsb\" (UniqueName: \"kubernetes.io/projected/01191d10-8c6a-4454-a048-ebc1b9cd3593-kube-api-access-g6dsb\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.666527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.666491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/01191d10-8c6a-4454-a048-ebc1b9cd3593-config-file\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.666527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.666527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6dsb\" (UniqueName: \"kubernetes.io/projected/01191d10-8c6a-4454-a048-ebc1b9cd3593-kube-api-access-g6dsb\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.667244 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.667220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/01191d10-8c6a-4454-a048-ebc1b9cd3593-config-file\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.678785 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.678755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6dsb\" (UniqueName: \"kubernetes.io/projected/01191d10-8c6a-4454-a048-ebc1b9cd3593-kube-api-access-g6dsb\") pod \"limitador-limitador-54d6d988d5-l76pr\" (UID: \"01191d10-8c6a-4454-a048-ebc1b9cd3593\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.794118 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.794000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:38.925903 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:38.925869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:21:38.929769 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:21:38.929742    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01191d10_8c6a_4454_a048_ebc1b9cd3593.slice/crio-6b766b97f5106681c096ccb35a9d7a809b6ecc5e5326b2fdaea5f875502a37e6 WatchSource:0}: Error finding container 6b766b97f5106681c096ccb35a9d7a809b6ecc5e5326b2fdaea5f875502a37e6: Status 404 returned error can't find the container with id 6b766b97f5106681c096ccb35a9d7a809b6ecc5e5326b2fdaea5f875502a37e6
Jul 23 11:21:39.152129 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:39.152024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr" event={"ID":"01191d10-8c6a-4454-a048-ebc1b9cd3593","Type":"ContainerStarted","Data":"6b766b97f5106681c096ccb35a9d7a809b6ecc5e5326b2fdaea5f875502a37e6"}
Jul 23 11:21:42.165663 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:42.165623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr" event={"ID":"01191d10-8c6a-4454-a048-ebc1b9cd3593","Type":"ContainerStarted","Data":"fe2076fe92453848ed020febee3938fce9d27f91d642c9d12fadcd88a3e1f43e"}
Jul 23 11:21:42.166034 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:42.165748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:21:42.183590 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:42.183544    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr" podStartSLOduration=1.6000300429999998 podStartE2EDuration="4.183530344s" totalImagesPullingTime="2.583500301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:21:42.183112646 +0000 UTC m=+676.015920873" watchObservedRunningTime="2026-07-23 11:21:42.183530344 +0000 UTC m=+676.016338612"
Jul 23 11:21:53.169653 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:21:53.169615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l76pr"
Jul 23 11:22:57.660359 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:22:57.660325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:23:14.370334 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:14.370299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:23:26.783138 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:26.783087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:23:41.601295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.601259    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"]
Jul 23 11:23:41.606888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.606848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.611231 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.611207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9687v\""
Jul 23 11:23:41.611380 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.611207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 11:23:41.611380 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.611212    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 11:23:41.611380 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.611372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 11:23:41.615947 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.615920    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"]
Jul 23 11:23:41.705685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.705685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c15a2-d7b6-40d8-8735-4328888d524f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.705901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.705901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.705901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmdb\" (UniqueName: \"kubernetes.io/projected/714c15a2-d7b6-40d8-8735-4328888d524f-kube-api-access-dpmdb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.705901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.705865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807211 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807211 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c15a2-d7b6-40d8-8735-4328888d524f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmdb\" (UniqueName: \"kubernetes.io/projected/714c15a2-d7b6-40d8-8735-4328888d524f-kube-api-access-dpmdb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807820 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807820 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.807820 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.807716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.809680 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.809660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714c15a2-d7b6-40d8-8735-4328888d524f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.809937 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.809917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714c15a2-d7b6-40d8-8735-4328888d524f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.819945 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.819913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmdb\" (UniqueName: \"kubernetes.io/projected/714c15a2-d7b6-40d8-8735-4328888d524f-kube-api-access-dpmdb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql4hh\" (UID: \"714c15a2-d7b6-40d8-8735-4328888d524f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:41.918156 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:41.918054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:42.049659 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:42.049626    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"]
Jul 23 11:23:42.053242 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:23:42.053212    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714c15a2_d7b6_40d8_8735_4328888d524f.slice/crio-a77af4f0c1dff10c65b2733dbaff931551e5819a342b50ac4e7c337ab3be9ab7 WatchSource:0}: Error finding container a77af4f0c1dff10c65b2733dbaff931551e5819a342b50ac4e7c337ab3be9ab7: Status 404 returned error can't find the container with id a77af4f0c1dff10c65b2733dbaff931551e5819a342b50ac4e7c337ab3be9ab7
Jul 23 11:23:42.054978 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:42.054952    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:23:42.617609 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:42.617558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh" event={"ID":"714c15a2-d7b6-40d8-8735-4328888d524f","Type":"ContainerStarted","Data":"a77af4f0c1dff10c65b2733dbaff931551e5819a342b50ac4e7c337ab3be9ab7"}
Jul 23 11:23:42.689825 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:42.689791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:23:43.623240 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:43.623152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh" event={"ID":"714c15a2-d7b6-40d8-8735-4328888d524f","Type":"ContainerStarted","Data":"4a84ed39c79ad20b347982a1d0c9e15e8582ffa6da1b0b38d7431533ddc844d7"}
Jul 23 11:23:43.643948 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:43.643898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh" podStartSLOduration=1.515475576 podStartE2EDuration="2.643878692s" totalImagesPullingTime="1.128403116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:23:43.642493606 +0000 UTC m=+797.475301869" watchObservedRunningTime="2026-07-23 11:23:43.643878692 +0000 UTC m=+797.476686924"
Jul 23 11:23:51.918960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:51.918875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:51.918960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:51.918947    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:51.931442 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:51.931417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:52.670987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:52.670958    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql4hh"
Jul 23 11:23:53.791051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:53.791019    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:23:59.336008 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:23:59.335970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:24:10.781217 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:24:10.781183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:25:46.297520 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:25:46.297484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:25:57.191493 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:25:57.191463    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:26:06.587527 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:26:06.587487    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:27:48.598485 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:27:48.598449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:28:00.585442 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:28:00.585407    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:28:34.115961 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:28:34.115926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:28:43.178256 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:28:43.178226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:28:50.586427 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:28:50.586392    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:28:57.892526 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:28:57.892487    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:29:08.898283 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:29:08.898244    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:29:20.099123 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:29:20.099076    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:29:31.388587 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:29:31.388548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:29:41.186672 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:29:41.186638    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:30:04.081713 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:30:04.081633    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:30:15.679501 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:30:15.679462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:30:37.889165 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:30:37.889130    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:30:49.096843 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:30:49.096808    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:31:05.479771 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:31:05.479730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:31:14.881064 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:31:14.881025    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:31:25.086777 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:31:25.086746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:31:35.188499 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:31:35.188462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:31:45.281620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:31:45.281581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:33:31.675003 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:33:31.674964    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:33:43.787636 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:33:43.787600    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:33:52.990922 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:33:52.990875    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:04.491463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:04.491429    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:16.684479 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:16.684442    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:26.781327 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:26.781290    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:34.781419 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:34.781388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:46.483206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:46.483169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:34:57.587295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:34:57.587260    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:35:05.187723 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:35:05.187687    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:35:17.777597 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:35:17.777558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:35:26.786355 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:35:26.786325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:35:44.281752 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:35:44.281711    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:35:53.690529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:35:53.690444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:36:08.392207 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:36:08.392170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:36:21.488036 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:36:21.487987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:36:55.591316 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:36:55.591278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:37:10.188399 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:37:10.188364    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:37:17.978398 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:37:17.978307    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:37:34.481910 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:37:34.481874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:37:44.285757 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:37:44.285721    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:37:53.484689 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:37:53.484654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:38:03.887819 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:38:03.887778    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:38:13.103685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:38:13.103646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:38:22.893772 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:38:22.893737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:38:33.919956 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:38:33.919923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:38:52.687208 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:38:52.687129    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:39:04.416861 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:39:04.416823    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:39:16.692464 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:39:16.692430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:39:25.662028 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:39:25.661988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:39:37.499638 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:39:37.499600    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:39:49.387657 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:39:49.387618    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:40:00.482989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:00.482956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:40:09.999615 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:09.999580    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:40:33.485728 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:33.485695    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:40:41.642731 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.642700    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:40:41.645995 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.645974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.648462 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.648436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9729caf1-serving-cert\""
Jul 23 11:40:41.648577 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.648494    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:40:41.648577 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.648533    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:40:41.649572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.649555    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:40:41.712501 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.712468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:40:41.747381 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.747351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.747506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.747388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dxp\" (UniqueName: \"kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.747506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.747457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.848409 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.848382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.848525 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.848426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.848525 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.848450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dxp\" (UniqueName: \"kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.848593 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:40:41.848544    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-serving-cert: secret "maas-api-e2e-ait-9729caf1-serving-cert" not found
Jul 23 11:40:41.848648 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:40:41.848636    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls podName:bbc63ec5-0a02-469e-a0b8-088e4ed0fe86 nodeName:}" failed. No retries permitted until 2026-07-23 11:40:42.348614983 +0000 UTC m=+1816.181423197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls") pod "maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" (UID: "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86") : secret "maas-api-e2e-ait-9729caf1-serving-cert" not found
Jul 23 11:40:41.849057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.849039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:41.863346 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:41.863316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dxp\" (UniqueName: \"kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:42.353444 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:42.353406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:42.355995 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:42.355973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") pod \"maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:42.555949 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:42.555915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:42.686716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:42.686694    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:40:42.689083 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:40:42.689054    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc63ec5_0a02_469e_a0b8_088e4ed0fe86.slice/crio-849680f9d09108fdec1f374ee8fd4f6db9f040c00307540759e9b8d1a3b11830 WatchSource:0}: Error finding container 849680f9d09108fdec1f374ee8fd4f6db9f040c00307540759e9b8d1a3b11830: Status 404 returned error can't find the container with id 849680f9d09108fdec1f374ee8fd4f6db9f040c00307540759e9b8d1a3b11830
Jul 23 11:40:42.690496 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:42.690475    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:40:43.482391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:43.482350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" event={"ID":"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86","Type":"ContainerStarted","Data":"849680f9d09108fdec1f374ee8fd4f6db9f040c00307540759e9b8d1a3b11830"}
Jul 23 11:40:44.782443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:44.782405    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:40:45.493316 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:45.493280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" event={"ID":"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86","Type":"ContainerStarted","Data":"c3f6c1e5bd292304d742f271208c357730a881509b6b281dd3b222cb003892f4"}
Jul 23 11:40:45.493467 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:45.493361    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:40:45.511425 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:45.511374    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" podStartSLOduration=1.9290920470000001 podStartE2EDuration="4.511358112s" totalImagesPullingTime="2.582266065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:40:45.510241624 +0000 UTC m=+1819.343049860" watchObservedRunningTime="2026-07-23 11:40:45.511358112 +0000 UTC m=+1819.344166343"
Jul 23 11:40:51.503501 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:40:51.503472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:41:07.005775 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.005738    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"]
Jul 23 11:41:07.014105 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.014063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:07.017068 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.017035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wx4jb\""
Jul 23 11:41:07.017407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.017386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"]
Jul 23 11:41:07.054509 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.054481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9729caf1-7rz29\" (UID: \"0793e990-48d1-4b05-a18e-fe5d6cf12aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:07.155675 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.155639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9729caf1-7rz29\" (UID: \"0793e990-48d1-4b05-a18e-fe5d6cf12aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:07.156283 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.156265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9729caf1-7rz29\" (UID: \"0793e990-48d1-4b05-a18e-fe5d6cf12aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:07.325014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.324932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:07.456179 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.456135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"]
Jul 23 11:41:07.459689 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:41:07.459661    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0793e990_48d1_4b05_a18e_fe5d6cf12aac.slice/crio-03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19 WatchSource:0}: Error finding container 03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19: Status 404 returned error can't find the container with id 03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19
Jul 23 11:41:07.585159 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:07.585060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29" event={"ID":"0793e990-48d1-4b05-a18e-fe5d6cf12aac","Type":"ContainerStarted","Data":"03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19"}
Jul 23 11:41:10.600568 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:10.600479    2583 generic.go:356] "Generic (PLEG): container finished" podID="0793e990-48d1-4b05-a18e-fe5d6cf12aac" containerID="5ca118489152ce4c119b264666ab94ae5032d3c5003cb82b786218c1044e6011" exitCode=0
Jul 23 11:41:10.600568 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:10.600537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29" event={"ID":"0793e990-48d1-4b05-a18e-fe5d6cf12aac","Type":"ContainerDied","Data":"5ca118489152ce4c119b264666ab94ae5032d3c5003cb82b786218c1044e6011"}
Jul 23 11:41:11.737260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:11.737236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:11.796682 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:11.796648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca\") pod \"0793e990-48d1-4b05-a18e-fe5d6cf12aac\" (UID: \"0793e990-48d1-4b05-a18e-fe5d6cf12aac\") "
Jul 23 11:41:11.796990 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:11.796968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0793e990-48d1-4b05-a18e-fe5d6cf12aac" (UID: "0793e990-48d1-4b05-a18e-fe5d6cf12aac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:41:11.898168 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:11.898080    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0793e990-48d1-4b05-a18e-fe5d6cf12aac-maas-api-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:41:12.609650 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:12.609613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29" event={"ID":"0793e990-48d1-4b05-a18e-fe5d6cf12aac","Type":"ContainerDied","Data":"03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19"}
Jul 23 11:41:12.609797 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:12.609663    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ff10642edf5f2b64a76a0c70ef942c4b443949b14301cf782ad55b4dbf0d19"
Jul 23 11:41:12.609797 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:12.609628    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"
Jul 23 11:41:17.476585 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.476546    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:41:17.477042 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.476795    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" podUID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" containerName="maas-api" containerID="cri-o://c3f6c1e5bd292304d742f271208c357730a881509b6b281dd3b222cb003892f4" gracePeriod=30
Jul 23 11:41:17.633130 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.633070    2583 generic.go:356] "Generic (PLEG): container finished" podID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" containerID="c3f6c1e5bd292304d742f271208c357730a881509b6b281dd3b222cb003892f4" exitCode=0
Jul 23 11:41:17.633130 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.633117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" event={"ID":"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86","Type":"ContainerDied","Data":"c3f6c1e5bd292304d742f271208c357730a881509b6b281dd3b222cb003892f4"}
Jul 23 11:41:17.723399 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.723371    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:41:17.851545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.851442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca\") pod \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") "
Jul 23 11:41:17.851545 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.851500    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") pod \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") "
Jul 23 11:41:17.851830 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.851554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5dxp\" (UniqueName: \"kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp\") pod \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\" (UID: \"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86\") "
Jul 23 11:41:17.851899 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.851874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" (UID: "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:41:17.853881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.853855    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" (UID: "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:41:17.853999 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.853913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp" (OuterVolumeSpecName: "kube-api-access-b5dxp") pod "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" (UID: "bbc63ec5-0a02-469e-a0b8-088e4ed0fe86"). InnerVolumeSpecName "kube-api-access-b5dxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:41:17.953077 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.953044    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:41:17.953077 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.953071    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:41:17.953077 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:17.953081    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5dxp\" (UniqueName: \"kubernetes.io/projected/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86-kube-api-access-b5dxp\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:41:18.638254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.638220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p" event={"ID":"bbc63ec5-0a02-469e-a0b8-088e4ed0fe86","Type":"ContainerDied","Data":"849680f9d09108fdec1f374ee8fd4f6db9f040c00307540759e9b8d1a3b11830"}
Jul 23 11:41:18.638708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.638269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"
Jul 23 11:41:18.638708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.638280    2583 scope.go:117] "RemoveContainer" containerID="c3f6c1e5bd292304d742f271208c357730a881509b6b281dd3b222cb003892f4"
Jul 23 11:41:18.658878 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.658843    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:41:18.664358 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.664336    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9729caf1-869f8f98bf-pwf9p"]
Jul 23 11:41:18.744804 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:18.744779    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" path="/var/lib/kubelet/pods/bbc63ec5-0a02-469e-a0b8-088e4ed0fe86/volumes"
Jul 23 11:41:20.687068 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:20.687033    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:41:22.110559 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:22.110521    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"]
Jul 23 11:41:22.112908 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:22.112878    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9729caf1-7rz29"]
Jul 23 11:41:22.743950 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:22.743916    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0793e990-48d1-4b05-a18e-fe5d6cf12aac" path="/var/lib/kubelet/pods/0793e990-48d1-4b05-a18e-fe5d6cf12aac/volumes"
Jul 23 11:41:23.259160 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259125    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259407    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" containerName="maas-api"
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259418    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" containerName="maas-api"
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259426    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0793e990-48d1-4b05-a18e-fe5d6cf12aac" containerName="revoke-keys"
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259431    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0793e990-48d1-4b05-a18e-fe5d6cf12aac" containerName="revoke-keys"
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259483    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0793e990-48d1-4b05-a18e-fe5d6cf12aac" containerName="revoke-keys"
Jul 23 11:41:23.259515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.259492    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc63ec5-0a02-469e-a0b8-088e4ed0fe86" containerName="maas-api"
Jul 23 11:41:23.263956 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.263934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.266585 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.266565    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cde04e19-gw-openshift-default-dockercfg-llhkx\""
Jul 23 11:41:23.278915 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.278890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:41:23.394076 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394290 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394290 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27759\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394290 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394290 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.394452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.394420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495503 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495503 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495733 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495890 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495890 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27759\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.495987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.495964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.496218 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.496192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.496274 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.496216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.496332 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.496315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.498063 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.498039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.498413 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.498396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.504840 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.504808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.505047 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.505026    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27759\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759\") pod \"e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") " pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.573550 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.573459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:23.712255 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.712228    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:41:23.714774 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:41:23.714744    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeb50bf_bf14_48c3_8cfc_9b9c717b128e.slice/crio-65cea5d81dfccc767a34ae5b42c95a7c64ad3b73836a029c9cecff6b711f380e WatchSource:0}: Error finding container 65cea5d81dfccc767a34ae5b42c95a7c64ad3b73836a029c9cecff6b711f380e: Status 404 returned error can't find the container with id 65cea5d81dfccc767a34ae5b42c95a7c64ad3b73836a029c9cecff6b711f380e
Jul 23 11:41:23.716747 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.716716    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:23.716837 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.716784    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:23.716837 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:23.716814    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:24.665506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:24.665469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerStarted","Data":"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"}
Jul 23 11:41:24.665882 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:24.665522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerStarted","Data":"65cea5d81dfccc767a34ae5b42c95a7c64ad3b73836a029c9cecff6b711f380e"}
Jul 23 11:41:24.686970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:24.686918    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podStartSLOduration=1.6869027779999999 podStartE2EDuration="1.686902778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:24.685249125 +0000 UTC m=+1858.518057369" watchObservedRunningTime="2026-07-23 11:41:24.686902778 +0000 UTC m=+1858.519711011"
Jul 23 11:41:25.573907 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:25.573871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:25.578828 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:25.578805    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:25.670144 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:25.670112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:25.671030 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:25.671009    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:26.224204 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.224170    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:41:26.227342 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.227324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.230181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.230155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cde04e19-serving-cert\""
Jul 23 11:41:26.230181 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.230169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:41:26.230327 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.230172    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:41:26.231282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.231265    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:41:26.239624 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.239605    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:41:26.322013 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.321980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4tg\" (UniqueName: \"kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.322246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.322024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.322246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.322156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.423602 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.423566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.423878 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.423860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4tg\" (UniqueName: \"kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.424026 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.424012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.424613 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.424580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.428117 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.428066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.434002 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.433979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4tg\" (UniqueName: \"kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg\") pod \"maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.540718 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.540648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:41:26.548201 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.548177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:26.691420 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:26.691388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:41:26.694823 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:41:26.694798    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50a6acf_61c3_4acb_9719_c26a1f003e79.slice/crio-7c1f41f54df2e5df1b4f6a5b2c1528db683352eb97baffdbf489117000ac52dd WatchSource:0}: Error finding container 7c1f41f54df2e5df1b4f6a5b2c1528db683352eb97baffdbf489117000ac52dd: Status 404 returned error can't find the container with id 7c1f41f54df2e5df1b4f6a5b2c1528db683352eb97baffdbf489117000ac52dd
Jul 23 11:41:27.679431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:27.679390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" event={"ID":"e50a6acf-61c3-4acb-9719-c26a1f003e79","Type":"ContainerStarted","Data":"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"}
Jul 23 11:41:27.679672 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:27.679649    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:27.679719 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:27.679697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" event={"ID":"e50a6acf-61c3-4acb-9719-c26a1f003e79","Type":"ContainerStarted","Data":"7c1f41f54df2e5df1b4f6a5b2c1528db683352eb97baffdbf489117000ac52dd"}
Jul 23 11:41:27.698019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:27.697974    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" podStartSLOduration=1.277856534 podStartE2EDuration="1.69795554s" totalImagesPullingTime="420.099006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:27.696496169 +0000 UTC m=+1861.529304425" watchObservedRunningTime="2026-07-23 11:41:27.69795554 +0000 UTC m=+1861.530763773"
Jul 23 11:41:31.582629 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:31.582593    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:41:33.689885 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:33.689851    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:41:39.734901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.734867    2583 generic.go:356] "Generic (PLEG): container finished" podID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerID="d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b" exitCode=0
Jul 23 11:41:39.735392 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.734934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerDied","Data":"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"}
Jul 23 11:41:39.735392 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.735324    2583 scope.go:117] "RemoveContainer" containerID="d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"
Jul 23 11:41:39.736473 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.736447    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:39.736573 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.736527    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:39.736631 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:39.736572    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:41:40.670189 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:40.670152    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:40.750515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:40.750471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerStarted","Data":"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"}
Jul 23 11:41:41.574325 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:41.574279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:41.575539 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:41.575509    2583 patch_prober.go:28] interesting pod/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 23 11:41:41.575666 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:41.575565    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 23 11:41:42.574353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:42.574314    2583 patch_prober.go:28] interesting pod/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 23 11:41:42.574754 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:42.574377    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 23 11:41:43.574305 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:43.574264    2583 patch_prober.go:28] interesting pod/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 23 11:41:43.574510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:43.574329    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 23 11:41:44.573977 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:44.573941    2583 patch_prober.go:28] interesting pod/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Jul 23 11:41:44.574192 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:44.574005    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Jul 23 11:41:45.577846 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:45.577807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:45.578338 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:45.578208    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:41:45.578988 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:41:45.578965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:42:01.569626 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.569589    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:42:01.570610 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.569886    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" podUID="e50a6acf-61c3-4acb-9719-c26a1f003e79" containerName="maas-api" containerID="cri-o://ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b" gracePeriod=30
Jul 23 11:42:01.815572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.815547    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:42:01.829188 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.829119    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca\") pod \"e50a6acf-61c3-4acb-9719-c26a1f003e79\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") "
Jul 23 11:42:01.829188 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.829158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb4tg\" (UniqueName: \"kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg\") pod \"e50a6acf-61c3-4acb-9719-c26a1f003e79\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") "
Jul 23 11:42:01.829391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.829207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls\") pod \"e50a6acf-61c3-4acb-9719-c26a1f003e79\" (UID: \"e50a6acf-61c3-4acb-9719-c26a1f003e79\") "
Jul 23 11:42:01.829522 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.829497    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e50a6acf-61c3-4acb-9719-c26a1f003e79" (UID: "e50a6acf-61c3-4acb-9719-c26a1f003e79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:42:01.831384 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.831357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg" (OuterVolumeSpecName: "kube-api-access-xb4tg") pod "e50a6acf-61c3-4acb-9719-c26a1f003e79" (UID: "e50a6acf-61c3-4acb-9719-c26a1f003e79"). InnerVolumeSpecName "kube-api-access-xb4tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:42:01.831384 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.831364    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e50a6acf-61c3-4acb-9719-c26a1f003e79" (UID: "e50a6acf-61c3-4acb-9719-c26a1f003e79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:42:01.842752 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.842727    2583 generic.go:356] "Generic (PLEG): container finished" podID="e50a6acf-61c3-4acb-9719-c26a1f003e79" containerID="ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b" exitCode=0
Jul 23 11:42:01.842881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.842779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"
Jul 23 11:42:01.842881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.842802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" event={"ID":"e50a6acf-61c3-4acb-9719-c26a1f003e79","Type":"ContainerDied","Data":"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"}
Jul 23 11:42:01.842881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.842855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk" event={"ID":"e50a6acf-61c3-4acb-9719-c26a1f003e79","Type":"ContainerDied","Data":"7c1f41f54df2e5df1b4f6a5b2c1528db683352eb97baffdbf489117000ac52dd"}
Jul 23 11:42:01.842997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.842888    2583 scope.go:117] "RemoveContainer" containerID="ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"
Jul 23 11:42:01.852523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.852500    2583 scope.go:117] "RemoveContainer" containerID="ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"
Jul 23 11:42:01.852807 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:01.852781    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b\": container with ID starting with ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b not found: ID does not exist" containerID="ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"
Jul 23 11:42:01.852853 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.852817    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b"} err="failed to get container status \"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b\": rpc error: code = NotFound desc = could not find container \"ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b\": container with ID starting with ae65ca6190022c84eca41d8057bf73a4c0ce6c197cf66ff17f3d57bd23598a6b not found: ID does not exist"
Jul 23 11:42:01.864893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.864866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:42:01.867643 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.867616    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cde04e19-75bd96c547-8vwwk"]
Jul 23 11:42:01.930156 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.930126    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e50a6acf-61c3-4acb-9719-c26a1f003e79-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:01.930156 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.930152    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4tg\" (UniqueName: \"kubernetes.io/projected/e50a6acf-61c3-4acb-9719-c26a1f003e79-kube-api-access-xb4tg\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:01.930323 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:01.930164    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e50a6acf-61c3-4acb-9719-c26a1f003e79-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:02.745627 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:02.745585    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50a6acf-61c3-4acb-9719-c26a1f003e79" path="/var/lib/kubelet/pods/e50a6acf-61c3-4acb-9719-c26a1f003e79/volumes"
Jul 23 11:42:04.778074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:04.778032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:42:06.376022 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:06.375990    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:42:06.376458 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:06.376226    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy" containerID="cri-o://14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99" gracePeriod=30
Jul 23 11:42:10.125790 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.125756    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:10.126233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.126078    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50a6acf-61c3-4acb-9719-c26a1f003e79" containerName="maas-api"
Jul 23 11:42:10.126233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.126104    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50a6acf-61c3-4acb-9719-c26a1f003e79" containerName="maas-api"
Jul 23 11:42:10.126233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.126161    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50a6acf-61c3-4acb-9719-c26a1f003e79" containerName="maas-api"
Jul 23 11:42:10.130332 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.130311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.133561 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.133535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:42:10.133696 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.133561    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:42:10.133696 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.133566    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:42:10.133696 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.133547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b54bb921-serving-cert\""
Jul 23 11:42:10.142531 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.142506    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:10.199459 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.199417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvrx\" (UniqueName: \"kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.199641 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.199492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.199641 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.199633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.300499 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.300456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvrx\" (UniqueName: \"kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.300499 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.300513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.300773 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.300608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.300773 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:10.300694    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-serving-cert: secret "maas-api-e2e-ait-b54bb921-serving-cert" not found
Jul 23 11:42:10.300773 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:10.300764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls podName:3794af65-0925-40bb-b3a9-5430de5e5f3c nodeName:}" failed. No retries permitted until 2026-07-23 11:42:10.800746961 +0000 UTC m=+1904.633555174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls") pod "maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" (UID: "3794af65-0925-40bb-b3a9-5430de5e5f3c") : secret "maas-api-e2e-ait-b54bb921-serving-cert" not found
Jul 23 11:42:10.301279 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.301261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.309404 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.309380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvrx\" (UniqueName: \"kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.805427 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.805385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:10.808006 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:10.807972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") pod \"maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:11.040342 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.040302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:11.185950 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.185920    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:11.189866 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:42:11.189834    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3794af65_0925_40bb_b3a9_5430de5e5f3c.slice/crio-f159bd05815447745b97524c3cec8d3fbf6225d3f8788b9b789eb4bb8399ea04 WatchSource:0}: Error finding container f159bd05815447745b97524c3cec8d3fbf6225d3f8788b9b789eb4bb8399ea04: Status 404 returned error can't find the container with id f159bd05815447745b97524c3cec8d3fbf6225d3f8788b9b789eb4bb8399ea04
Jul 23 11:42:11.625794 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.625770    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:42:11.715050 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715149    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715201    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715225    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715282 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715249    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715292    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27759\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs\") pod \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\" (UID: \"deeb50bf-bf14-48c3-8cfc-9b9c717b128e\") "
Jul 23 11:42:11.715661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:42:11.715661 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715598    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.715866 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:42:11.715866 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:42:11.715961 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715904    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:42:11.715961 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.715914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data" (OuterVolumeSpecName: "istio-data") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:42:11.717848 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.717822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:42:11.718148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.718121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759" (OuterVolumeSpecName: "kube-api-access-27759") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "kube-api-access-27759". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:42:11.718148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.718127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token" (OuterVolumeSpecName: "istio-token") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:42:11.718291 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.718152    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "deeb50bf-bf14-48c3-8cfc-9b9c717b128e" (UID: "deeb50bf-bf14-48c3-8cfc-9b9c717b128e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:42:11.815989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.815955    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.815989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.815986    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.815989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.815995    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.816238 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.816003    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.816238 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.816012    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.816238 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.816020    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.816238 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.816028    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27759\" (UniqueName: \"kubernetes.io/projected/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-kube-api-access-27759\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.816238 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.816036    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeb50bf-bf14-48c3-8cfc-9b9c717b128e-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:11.889577 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.889526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" event={"ID":"3794af65-0925-40bb-b3a9-5430de5e5f3c","Type":"ContainerStarted","Data":"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"}
Jul 23 11:42:11.889759 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.889607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" event={"ID":"3794af65-0925-40bb-b3a9-5430de5e5f3c","Type":"ContainerStarted","Data":"f159bd05815447745b97524c3cec8d3fbf6225d3f8788b9b789eb4bb8399ea04"}
Jul 23 11:42:11.889759 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.889640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:11.891149 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.891128    2583 generic.go:356] "Generic (PLEG): container finished" podID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerID="14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99" exitCode=0
Jul 23 11:42:11.891265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.891161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerDied","Data":"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"}
Jul 23 11:42:11.891265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.891182    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"
Jul 23 11:42:11.891265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.891219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29" event={"ID":"deeb50bf-bf14-48c3-8cfc-9b9c717b128e","Type":"ContainerDied","Data":"65cea5d81dfccc767a34ae5b42c95a7c64ad3b73836a029c9cecff6b711f380e"}
Jul 23 11:42:11.891265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.891250    2583 scope.go:117] "RemoveContainer" containerID="14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"
Jul 23 11:42:11.900501 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.900481    2583 scope.go:117] "RemoveContainer" containerID="d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"
Jul 23 11:42:11.908682 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.908665    2583 scope.go:117] "RemoveContainer" containerID="14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"
Jul 23 11:42:11.908969 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:11.908947    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99\": container with ID starting with 14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99 not found: ID does not exist" containerID="14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"
Jul 23 11:42:11.909073 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.908976    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99"} err="failed to get container status \"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99\": rpc error: code = NotFound desc = could not find container \"14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99\": container with ID starting with 14491dd60b5703eca76f6ac05c173790d394a3c681cdd7031d9a0ca8b3c70c99 not found: ID does not exist"
Jul 23 11:42:11.909073 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.908994    2583 scope.go:117] "RemoveContainer" containerID="d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"
Jul 23 11:42:11.909309 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:11.909286    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b\": container with ID starting with d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b not found: ID does not exist" containerID="d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"
Jul 23 11:42:11.909396 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.909314    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b"} err="failed to get container status \"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b\": rpc error: code = NotFound desc = could not find container \"d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b\": container with ID starting with d41a84c8c69bf50e5727601dfcc13ae65a4300fd89ea1d9e245e47b82db21e5b not found: ID does not exist"
Jul 23 11:42:11.920154 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.920088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" podStartSLOduration=1.573143171 podStartE2EDuration="1.920072185s" totalImagesPullingTime="346.929014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:11.90643154 +0000 UTC m=+1905.739239793" watchObservedRunningTime="2026-07-23 11:42:11.920072185 +0000 UTC m=+1905.752880446"
Jul 23 11:42:11.920292 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.920233    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:42:11.925180 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:11.925153    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cde04e19-gw-openshift-default-94f9ffb67-jbr29"]
Jul 23 11:42:12.745482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:12.745449    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" path="/var/lib/kubelet/pods/deeb50bf-bf14-48c3-8cfc-9b9c717b128e/volumes"
Jul 23 11:42:15.791436 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:15.791391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:42:17.902225 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:17.902186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:35.716056 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716020    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"]
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716313    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716324    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716396    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.716521 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.716404    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="deeb50bf-bf14-48c3-8cfc-9b9c717b128e" containerName="istio-proxy"
Jul 23 11:42:35.719067 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.719049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:35.722648 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.722618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wx4jb\""
Jul 23 11:42:35.727059 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.727019    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"]
Jul 23 11:42:35.823893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.823853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b54bb921-r8thc\" (UID: \"5070948e-0e18-48a0-bced-4b58d70b7847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:35.925307 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.925271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b54bb921-r8thc\" (UID: \"5070948e-0e18-48a0-bced-4b58d70b7847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:35.925897 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:35.925878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b54bb921-r8thc\" (UID: \"5070948e-0e18-48a0-bced-4b58d70b7847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:36.029539 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:36.029442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:36.164403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:36.164380    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"]
Jul 23 11:42:36.166443 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:42:36.166421    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5070948e_0e18_48a0_bced_4b58d70b7847.slice/crio-4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f WatchSource:0}: Error finding container 4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f: Status 404 returned error can't find the container with id 4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f
Jul 23 11:42:36.997463 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:36.997427    2583 generic.go:356] "Generic (PLEG): container finished" podID="5070948e-0e18-48a0-bced-4b58d70b7847" containerID="85c2bbd2c8c0952c211ed23b3233a0b615d976fdeb0c365f315ecc14aa613059" exitCode=0
Jul 23 11:42:36.997793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:36.997473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc" event={"ID":"5070948e-0e18-48a0-bced-4b58d70b7847","Type":"ContainerDied","Data":"85c2bbd2c8c0952c211ed23b3233a0b615d976fdeb0c365f315ecc14aa613059"}
Jul 23 11:42:36.997793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:36.997512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc" event={"ID":"5070948e-0e18-48a0-bced-4b58d70b7847","Type":"ContainerStarted","Data":"4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f"}
Jul 23 11:42:38.140653 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:38.140623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:38.245076 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:38.245034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca\") pod \"5070948e-0e18-48a0-bced-4b58d70b7847\" (UID: \"5070948e-0e18-48a0-bced-4b58d70b7847\") "
Jul 23 11:42:38.245444 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:38.245416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5070948e-0e18-48a0-bced-4b58d70b7847" (UID: "5070948e-0e18-48a0-bced-4b58d70b7847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:42:38.346337 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:38.346263    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5070948e-0e18-48a0-bced-4b58d70b7847-maas-api-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:39.008372 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:39.008287    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"
Jul 23 11:42:39.008529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:39.008285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc" event={"ID":"5070948e-0e18-48a0-bced-4b58d70b7847","Type":"ContainerDied","Data":"4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f"}
Jul 23 11:42:39.008529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:39.008421    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e90f0752fa263276a51de213451212d5e080c567db12f3770f6e0e52886394f"
Jul 23 11:42:46.152315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.152278    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:46.153072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.153036    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" podUID="3794af65-0925-40bb-b3a9-5430de5e5f3c" containerName="maas-api" containerID="cri-o://dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c" gracePeriod=30
Jul 23 11:42:46.406898 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.406834    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:46.407890 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.407873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca\") pod \"3794af65-0925-40bb-b3a9-5430de5e5f3c\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") "
Jul 23 11:42:46.407941 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.407912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") pod \"3794af65-0925-40bb-b3a9-5430de5e5f3c\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") "
Jul 23 11:42:46.407975 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.407939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxvrx\" (UniqueName: \"kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx\") pod \"3794af65-0925-40bb-b3a9-5430de5e5f3c\" (UID: \"3794af65-0925-40bb-b3a9-5430de5e5f3c\") "
Jul 23 11:42:46.408278 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.408259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3794af65-0925-40bb-b3a9-5430de5e5f3c" (UID: "3794af65-0925-40bb-b3a9-5430de5e5f3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:42:46.410191 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.410168    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3794af65-0925-40bb-b3a9-5430de5e5f3c" (UID: "3794af65-0925-40bb-b3a9-5430de5e5f3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:42:46.410285 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.410226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx" (OuterVolumeSpecName: "kube-api-access-nxvrx") pod "3794af65-0925-40bb-b3a9-5430de5e5f3c" (UID: "3794af65-0925-40bb-b3a9-5430de5e5f3c"). InnerVolumeSpecName "kube-api-access-nxvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:42:46.508761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.508728    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3794af65-0925-40bb-b3a9-5430de5e5f3c-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:46.508761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.508759    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3794af65-0925-40bb-b3a9-5430de5e5f3c-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:46.508956 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:46.508771    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvrx\" (UniqueName: \"kubernetes.io/projected/3794af65-0925-40bb-b3a9-5430de5e5f3c-kube-api-access-nxvrx\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:42:47.043518 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.043421    2583 generic.go:356] "Generic (PLEG): container finished" podID="3794af65-0925-40bb-b3a9-5430de5e5f3c" containerID="dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c" exitCode=0
Jul 23 11:42:47.043518 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.043485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" event={"ID":"3794af65-0925-40bb-b3a9-5430de5e5f3c","Type":"ContainerDied","Data":"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"}
Jul 23 11:42:47.043716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.043531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv" event={"ID":"3794af65-0925-40bb-b3a9-5430de5e5f3c","Type":"ContainerDied","Data":"f159bd05815447745b97524c3cec8d3fbf6225d3f8788b9b789eb4bb8399ea04"}
Jul 23 11:42:47.043716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.043551    2583 scope.go:117] "RemoveContainer" containerID="dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"
Jul 23 11:42:47.043716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.043487    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"
Jul 23 11:42:47.051796 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.051775    2583 scope.go:117] "RemoveContainer" containerID="dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"
Jul 23 11:42:47.052078 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:47.052054    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c\": container with ID starting with dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c not found: ID does not exist" containerID="dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"
Jul 23 11:42:47.052206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.052084    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c"} err="failed to get container status \"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c\": rpc error: code = NotFound desc = could not find container \"dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c\": container with ID starting with dfda1788eda72ff5606f750e9ce3003add7b8da238c3eb76c088bdbb4ac5d56c not found: ID does not exist"
Jul 23 11:42:47.060414 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.060390    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:47.064935 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:47.064912    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b54bb921-54555df5c8-bq7sv"]
Jul 23 11:42:48.744499 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:48.744463    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3794af65-0925-40bb-b3a9-5430de5e5f3c" path="/var/lib/kubelet/pods/3794af65-0925-40bb-b3a9-5430de5e5f3c/volumes"
Jul 23 11:42:49.480684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:49.480646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:42:50.797260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:50.797203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"]
Jul 23 11:42:50.798838 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:50.798814    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b54bb921-r8thc"]
Jul 23 11:42:52.690300 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690262    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:42:52.690670 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3794af65-0925-40bb-b3a9-5430de5e5f3c" containerName="maas-api"
Jul 23 11:42:52.690670 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690616    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3794af65-0925-40bb-b3a9-5430de5e5f3c" containerName="maas-api"
Jul 23 11:42:52.690670 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690628    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5070948e-0e18-48a0-bced-4b58d70b7847" containerName="revoke-keys"
Jul 23 11:42:52.690670 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690634    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5070948e-0e18-48a0-bced-4b58d70b7847" containerName="revoke-keys"
Jul 23 11:42:52.690805 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690682    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5070948e-0e18-48a0-bced-4b58d70b7847" containerName="revoke-keys"
Jul 23 11:42:52.690805 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.690691    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3794af65-0925-40bb-b3a9-5430de5e5f3c" containerName="maas-api"
Jul 23 11:42:52.693746 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.693727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.696544 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.696521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f70c9fb7-openshift-default-dockercfg-nlqd7\""
Jul 23 11:42:52.706783 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.706760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:42:52.743928 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.743893    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5070948e-0e18-48a0-bced-4b58d70b7847" path="/var/lib/kubelet/pods/5070948e-0e18-48a0-bced-4b58d70b7847/volumes"
Jul 23 11:42:52.755627 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755627 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jbx\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.755816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.756057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.756057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.755884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857109 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857260 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jbx\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857657 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857777 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857856 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.857945 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.858016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.857996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.859806 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.859783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.860021 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.859998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.866223 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.866191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:52.866316 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:52.866266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jbx\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx\") pod \"e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") " pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:53.004615 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:53.004524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:53.171371 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:53.171218    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:42:53.174257 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:42:53.174228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57862864_217e_43fa_8f87_1e334bc631b4.slice/crio-9e09ef5aba9186e1275ad1280e6e0d87b8047278330510c7890ea8b7de806f1f WatchSource:0}: Error finding container 9e09ef5aba9186e1275ad1280e6e0d87b8047278330510c7890ea8b7de806f1f: Status 404 returned error can't find the container with id 9e09ef5aba9186e1275ad1280e6e0d87b8047278330510c7890ea8b7de806f1f
Jul 23 11:42:53.176809 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:53.176772    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:42:53.176937 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:53.176864    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:42:53.176997 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:53.176933    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:42:54.089333 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:54.089296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerStarted","Data":"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"}
Jul 23 11:42:54.089741 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:54.089352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerStarted","Data":"9e09ef5aba9186e1275ad1280e6e0d87b8047278330510c7890ea8b7de806f1f"}
Jul 23 11:42:54.112422 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:54.112372    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podStartSLOduration=2.112355874 podStartE2EDuration="2.112355874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:54.110162441 +0000 UTC m=+1947.942970674" watchObservedRunningTime="2026-07-23 11:42:54.112355874 +0000 UTC m=+1947.945164107"
Jul 23 11:42:55.004876 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.004832    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:55.009882 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.009855    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:55.093844 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.093817    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:55.094844 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.094827    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:42:55.554966 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.554927    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:42:55.558121 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.558075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.560774 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.560729    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f70c9fb7-serving-cert\""
Jul 23 11:42:55.560893 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.560871    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:42:55.560959 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.560922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:42:55.561039 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.561018    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:42:55.568677 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.568646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:42:55.583202 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.583172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.583396 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.583228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrhp\" (UniqueName: \"kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.583396 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.583313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.683779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.683739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrhp\" (UniqueName: \"kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.683986 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.683801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.683986 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.683854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.683986 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:55.683961    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-serving-cert: secret "maas-api-e2e-derive-f70c9fb7-serving-cert" not found
Jul 23 11:42:55.684185 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:42:55.684030    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls podName:cee5f6c3-025e-44c4-9e8b-6321d62befdb nodeName:}" failed. No retries permitted until 2026-07-23 11:42:56.184010925 +0000 UTC m=+1950.016819141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls") pod "maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" (UID: "cee5f6c3-025e-44c4-9e8b-6321d62befdb") : secret "maas-api-e2e-derive-f70c9fb7-serving-cert" not found
Jul 23 11:42:55.684553 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.684535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:55.694664 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:55.694630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrhp\" (UniqueName: \"kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:56.188917 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:56.188877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:56.191506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:56.191480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") pod \"maas-api-e2e-derive-f70c9fb7-564798c456-2fn64\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:56.469694 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:56.469588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:56.605568 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:56.605539    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:42:56.608137 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:42:56.608109    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee5f6c3_025e_44c4_9e8b_6321d62befdb.slice/crio-a92fb1db5477ea52272044776f667d9ed68b32d734210c94b6a53eafcfcf0473 WatchSource:0}: Error finding container a92fb1db5477ea52272044776f667d9ed68b32d734210c94b6a53eafcfcf0473: Status 404 returned error can't find the container with id a92fb1db5477ea52272044776f667d9ed68b32d734210c94b6a53eafcfcf0473
Jul 23 11:42:57.106285 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:57.106252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" event={"ID":"cee5f6c3-025e-44c4-9e8b-6321d62befdb","Type":"ContainerStarted","Data":"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"}
Jul 23 11:42:57.106408 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:57.106318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" event={"ID":"cee5f6c3-025e-44c4-9e8b-6321d62befdb","Type":"ContainerStarted","Data":"a92fb1db5477ea52272044776f667d9ed68b32d734210c94b6a53eafcfcf0473"}
Jul 23 11:42:57.106481 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:57.106431    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:42:57.124605 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:42:57.124564    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" podStartSLOduration=1.693709511 podStartE2EDuration="2.124547843s" totalImagesPullingTime="430.838332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:57.123607886 +0000 UTC m=+1950.956416119" watchObservedRunningTime="2026-07-23 11:42:57.124547843 +0000 UTC m=+1950.957356081"
Jul 23 11:43:00.384242 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:00.384208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:43:03.115954 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:03.115917    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:43:09.168991 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.168958    2583 generic.go:356] "Generic (PLEG): container finished" podID="57862864-217e-43fa-8f87-1e334bc631b4" containerID="6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde" exitCode=0
Jul 23 11:43:09.169438 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.169021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerDied","Data":"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"}
Jul 23 11:43:09.169438 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.169411    2583 scope.go:117] "RemoveContainer" containerID="6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"
Jul 23 11:43:09.170536 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.170510    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:43:09.170592 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.170578    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:43:09.170626 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:09.170613    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:43:10.093886 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:10.093840    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:10.175307 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:10.175259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerStarted","Data":"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"}
Jul 23 11:43:11.005111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:11.005056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:11.006243 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:11.006214    2583 patch_prober.go:28] interesting pod/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 23 11:43:11.006367 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:11.006281    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 23 11:43:12.005212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:12.005167    2583 patch_prober.go:28] interesting pod/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 23 11:43:12.005685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:12.005243    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 23 11:43:13.005595 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:13.005553    2583 patch_prober.go:28] interesting pod/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 23 11:43:13.005988 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:13.005611    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 23 11:43:14.005741 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:14.005696    2583 patch_prober.go:28] interesting pod/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 23 11:43:14.006180 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:14.005769    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 23 11:43:15.009473 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:15.009443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:15.009877 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:15.009808    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:15.010564 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:15.010542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:30.879512 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:30.879473    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:43:30.879960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:30.879721    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" podUID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" containerName="maas-api" containerID="cri-o://cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf" gracePeriod=30
Jul 23 11:43:31.135984 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.135925    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:43:31.267179 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.267145    2583 generic.go:356] "Generic (PLEG): container finished" podID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" containerID="cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf" exitCode=0
Jul 23 11:43:31.267360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.267195    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"
Jul 23 11:43:31.267360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.267233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" event={"ID":"cee5f6c3-025e-44c4-9e8b-6321d62befdb","Type":"ContainerDied","Data":"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"}
Jul 23 11:43:31.267360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.267288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64" event={"ID":"cee5f6c3-025e-44c4-9e8b-6321d62befdb","Type":"ContainerDied","Data":"a92fb1db5477ea52272044776f667d9ed68b32d734210c94b6a53eafcfcf0473"}
Jul 23 11:43:31.267360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.267312    2583 scope.go:117] "RemoveContainer" containerID="cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"
Jul 23 11:43:31.275623 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.275603    2583 scope.go:117] "RemoveContainer" containerID="cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"
Jul 23 11:43:31.275876 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:43:31.275857    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf\": container with ID starting with cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf not found: ID does not exist" containerID="cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"
Jul 23 11:43:31.275923 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.275887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf"} err="failed to get container status \"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf\": rpc error: code = NotFound desc = could not find container \"cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf\": container with ID starting with cb300d93a1bfb9c09ce63c581468168ef5a2efef019a98be6d45b8163924e6bf not found: ID does not exist"
Jul 23 11:43:31.315630 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.315592    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjrhp\" (UniqueName: \"kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp\") pod \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") "
Jul 23 11:43:31.315786 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.315673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") pod \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") "
Jul 23 11:43:31.315786 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.315756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca\") pod \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\" (UID: \"cee5f6c3-025e-44c4-9e8b-6321d62befdb\") "
Jul 23 11:43:31.316164 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.316130    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cee5f6c3-025e-44c4-9e8b-6321d62befdb" (UID: "cee5f6c3-025e-44c4-9e8b-6321d62befdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:43:31.317926 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.317900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp" (OuterVolumeSpecName: "kube-api-access-bjrhp") pod "cee5f6c3-025e-44c4-9e8b-6321d62befdb" (UID: "cee5f6c3-025e-44c4-9e8b-6321d62befdb"). InnerVolumeSpecName "kube-api-access-bjrhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:43:31.318036 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.317933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cee5f6c3-025e-44c4-9e8b-6321d62befdb" (UID: "cee5f6c3-025e-44c4-9e8b-6321d62befdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:43:31.416570 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.416476    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrhp\" (UniqueName: \"kubernetes.io/projected/cee5f6c3-025e-44c4-9e8b-6321d62befdb-kube-api-access-bjrhp\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:31.416570 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.416509    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee5f6c3-025e-44c4-9e8b-6321d62befdb-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:31.416570 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.416522    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee5f6c3-025e-44c4-9e8b-6321d62befdb-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:31.589151 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.589116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:43:31.595635 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:31.595604    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f70c9fb7-564798c456-2fn64"]
Jul 23 11:43:32.744815 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:32.744779    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" path="/var/lib/kubelet/pods/cee5f6c3-025e-44c4-9e8b-6321d62befdb/volumes"
Jul 23 11:43:33.786165 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:33.786126    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:43:35.540507 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:35.540469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:43:35.540968 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:35.540740    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy" containerID="cri-o://839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb" gracePeriod=30
Jul 23 11:43:40.798570 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.798542    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:40.897684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897585    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897761    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897795    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897840    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2jbx\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.897970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.897864    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy\") pod \"57862864-217e-43fa-8f87-1e334bc631b4\" (UID: \"57862864-217e-43fa-8f87-1e334bc631b4\") "
Jul 23 11:43:40.898289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898069    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:43:40.898289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:43:40.898289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898215    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.898455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:43:40.898455 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898364    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:43:40.898826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.898733    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:43:40.900151 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.900116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:43:40.900254 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.900234    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:43:40.900296 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.900246    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx" (OuterVolumeSpecName: "kube-api-access-f2jbx") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "kube-api-access-f2jbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:43:40.900491 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.900473    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57862864-217e-43fa-8f87-1e334bc631b4" (UID: "57862864-217e-43fa-8f87-1e334bc631b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:43:40.998761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998705    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.998761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998755    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jbx\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-kube-api-access-f2jbx\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.998761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998770    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.999019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998782    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.999019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998792    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57862864-217e-43fa-8f87-1e334bc631b4-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.999019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998803    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57862864-217e-43fa-8f87-1e334bc631b4-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.999019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998814    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57862864-217e-43fa-8f87-1e334bc631b4-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:40.999019 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:40.998824    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57862864-217e-43fa-8f87-1e334bc631b4-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:43:41.312113 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.312055    2583 generic.go:356] "Generic (PLEG): container finished" podID="57862864-217e-43fa-8f87-1e334bc631b4" containerID="839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb" exitCode=0
Jul 23 11:43:41.312310 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.312134    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"
Jul 23 11:43:41.312310 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.312134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerDied","Data":"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"}
Jul 23 11:43:41.312310 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.312192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t" event={"ID":"57862864-217e-43fa-8f87-1e334bc631b4","Type":"ContainerDied","Data":"9e09ef5aba9186e1275ad1280e6e0d87b8047278330510c7890ea8b7de806f1f"}
Jul 23 11:43:41.312310 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.312215    2583 scope.go:117] "RemoveContainer" containerID="839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"
Jul 23 11:43:41.321292 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.321272    2583 scope.go:117] "RemoveContainer" containerID="6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"
Jul 23 11:43:41.329405 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.329387    2583 scope.go:117] "RemoveContainer" containerID="839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"
Jul 23 11:43:41.329688 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:43:41.329668    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb\": container with ID starting with 839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb not found: ID does not exist" containerID="839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"
Jul 23 11:43:41.329740 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.329699    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb"} err="failed to get container status \"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb\": rpc error: code = NotFound desc = could not find container \"839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb\": container with ID starting with 839ffa49a4477c1fa1dfe71eadfb0b47ce8db71d791d4332dc73ebf9b48dddcb not found: ID does not exist"
Jul 23 11:43:41.329740 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.329719    2583 scope.go:117] "RemoveContainer" containerID="6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"
Jul 23 11:43:41.329989 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:43:41.329970    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde\": container with ID starting with 6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde not found: ID does not exist" containerID="6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"
Jul 23 11:43:41.330044 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.329996    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde"} err="failed to get container status \"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde\": rpc error: code = NotFound desc = could not find container \"6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde\": container with ID starting with 6b11de436c6183d7bd7824c0a048490e5a7b4535edff60b4d2dd89afbe751bde not found: ID does not exist"
Jul 23 11:43:41.334958 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.334933    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:43:41.338134 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:41.338082    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f70c9fb7-openshift-default-5dfd99499-tp74t"]
Jul 23 11:43:42.744841 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:43:42.744800    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57862864-217e-43fa-8f87-1e334bc631b4" path="/var/lib/kubelet/pods/57862864-217e-43fa-8f87-1e334bc631b4/volumes"
Jul 23 11:45:00.141947 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.141855    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"]
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142323    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142373    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" containerName="maas-api"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142381    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" containerName="maas-api"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142402    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142471    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee5f6c3-025e-44c4-9e8b-6321d62befdb" containerName="maas-api"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142483    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.142571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.142492    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="57862864-217e-43fa-8f87-1e334bc631b4" containerName="istio-proxy"
Jul 23 11:45:00.145409 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.145387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"
Jul 23 11:45:00.147984 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.147966    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wx4jb\""
Jul 23 11:45:00.153113 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.153069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"]
Jul 23 11:45:00.155515 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.155494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"
Jul 23 11:45:00.288844 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.288818    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"]
Jul 23 11:45:00.291233 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:45:00.291208    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ff0c31_956c_4da9_a65e_f42f511bd48a.slice/crio-1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1 WatchSource:0}: Error finding container 1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1: Status 404 returned error can't find the container with id 1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1
Jul 23 11:45:00.651244 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.651200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q" event={"ID":"b1ff0c31-956c-4da9-a65e-f42f511bd48a","Type":"ContainerStarted","Data":"1dc3f27b03ef147303df10a8f8d1c80962c695766fb381bee2bad2904304222f"}
Jul 23 11:45:00.651408 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.651266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q" event={"ID":"b1ff0c31-956c-4da9-a65e-f42f511bd48a","Type":"ContainerStarted","Data":"1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1"}
Jul 23 11:45:00.667127 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:00.667060    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q" podStartSLOduration=0.667044844 podStartE2EDuration="667.044844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:00.666388549 +0000 UTC m=+2074.499196781" watchObservedRunningTime="2026-07-23 11:45:00.667044844 +0000 UTC m=+2074.499853077"
Jul 23 11:45:01.657232 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:01.657195    2583 generic.go:356] "Generic (PLEG): container finished" podID="b1ff0c31-956c-4da9-a65e-f42f511bd48a" containerID="1dc3f27b03ef147303df10a8f8d1c80962c695766fb381bee2bad2904304222f" exitCode=0
Jul 23 11:45:01.657628 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:01.657281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q" event={"ID":"b1ff0c31-956c-4da9-a65e-f42f511bd48a","Type":"ContainerDied","Data":"1dc3f27b03ef147303df10a8f8d1c80962c695766fb381bee2bad2904304222f"}
Jul 23 11:45:01.677013 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:01.676984    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:45:02.795881 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:02.795857    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"
Jul 23 11:45:03.667145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:03.667082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q" event={"ID":"b1ff0c31-956c-4da9-a65e-f42f511bd48a","Type":"ContainerDied","Data":"1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1"}
Jul 23 11:45:03.667145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:03.667129    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746785-7q48q"
Jul 23 11:45:03.667403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:03.667158    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dfd76b9d6195f1ef093cd724d1a805d977efd96ce45cf96ae44850f21bbb6f1"
Jul 23 11:45:12.333505 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.333449    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:12.333894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.333754    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ff0c31-956c-4da9-a65e-f42f511bd48a" containerName="cleanup"
Jul 23 11:45:12.333894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.333768    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ff0c31-956c-4da9-a65e-f42f511bd48a" containerName="cleanup"
Jul 23 11:45:12.333894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.333835    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ff0c31-956c-4da9-a65e-f42f511bd48a" containerName="cleanup"
Jul 23 11:45:12.336749 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.336731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.339460 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.339439    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3403a9d3-openshift-default-dockercfg-m4476\""
Jul 23 11:45:12.353849 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.353826    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:12.364387 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364387 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7m97\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364485    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364802 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364802 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.364802 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.364629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465625 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7m97\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.465989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.465971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.466183 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.466162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.466248 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.466227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.466306 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.466289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.468057 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.468029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.468219 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.468204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.474151 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.474131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.474330 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.474310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7m97\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97\") pod \"e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") " pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.578885 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.578850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:45:12.647582 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.647487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:12.787571 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.787547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:12.789825 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:45:12.789788    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ab1667_ecb1_43f8_a17c_ab0ca1bf5376.slice/crio-d87e2c8cd452def699da11e5a9395aba5bc0cb29591caf8aa5f608d731a740cc WatchSource:0}: Error finding container d87e2c8cd452def699da11e5a9395aba5bc0cb29591caf8aa5f608d731a740cc: Status 404 returned error can't find the container with id d87e2c8cd452def699da11e5a9395aba5bc0cb29591caf8aa5f608d731a740cc
Jul 23 11:45:12.792332 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.792293    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:12.792402 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.792385    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:12.792442 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:12.792428    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:13.713247 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:13.713209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerStarted","Data":"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"}
Jul 23 11:45:13.713730 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:13.713261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerStarted","Data":"d87e2c8cd452def699da11e5a9395aba5bc0cb29591caf8aa5f608d731a740cc"}
Jul 23 11:45:13.735985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:13.735936    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podStartSLOduration=1.735919234 podStartE2EDuration="1.735919234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:13.734954748 +0000 UTC m=+2087.567762982" watchObservedRunningTime="2026-07-23 11:45:13.735919234 +0000 UTC m=+2087.568727467"
Jul 23 11:45:14.648005 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:14.647968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:14.653377 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:14.653343    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:14.717984 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:14.717952    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:14.719541 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:14.719520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:15.795970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.795914    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:15.799353 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.799331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.802205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.802163    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:45:15.802205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.802164    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3403a9d3-serving-cert\""
Jul 23 11:45:15.802371 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.802169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:45:15.803255 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.803219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:45:15.810915 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.810854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:15.893347 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.893302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5hf\" (UniqueName: \"kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.893546 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.893372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.893546 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.893480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.994558 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.994512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.994741 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.994603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.994741 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.994631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5hf\" (UniqueName: \"kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.995314 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.995294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:15.997153 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:15.997135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:16.008431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:16.008407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5hf\" (UniqueName: \"kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf\") pod \"maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:16.112233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:16.112133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:16.273716 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:16.273683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:16.274873 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:45:16.274849    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1445d4_7888_4f96_91d7_b5d0cc993ed1.slice/crio-6cd05ffcc35a3b4e985ea81c9e8bb3aaca68633fe5f072b8927d30bd825439bd WatchSource:0}: Error finding container 6cd05ffcc35a3b4e985ea81c9e8bb3aaca68633fe5f072b8927d30bd825439bd: Status 404 returned error can't find the container with id 6cd05ffcc35a3b4e985ea81c9e8bb3aaca68633fe5f072b8927d30bd825439bd
Jul 23 11:45:16.726980 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:16.726942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" event={"ID":"bc1445d4-7888-4f96-91d7-b5d0cc993ed1","Type":"ContainerStarted","Data":"6cd05ffcc35a3b4e985ea81c9e8bb3aaca68633fe5f072b8927d30bd825439bd"}
Jul 23 11:45:17.732802 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:17.732765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" event={"ID":"bc1445d4-7888-4f96-91d7-b5d0cc993ed1","Type":"ContainerStarted","Data":"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"}
Jul 23 11:45:17.733217 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:17.732905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:17.750345 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:17.750302    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" podStartSLOduration=1.473276604 podStartE2EDuration="2.750287431s" totalImagesPullingTime="1.277010827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:17.749835088 +0000 UTC m=+2091.582643323" watchObservedRunningTime="2026-07-23 11:45:17.750287431 +0000 UTC m=+2091.583095664"
Jul 23 11:45:21.189648 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:21.189614    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:45:23.750423 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:23.750392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:29.719709 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.719671    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 23 11:45:29.720131 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.719740    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 23 11:45:29.790778 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.790744    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerID="be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed" exitCode=0
Jul 23 11:45:29.790946 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.790818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerDied","Data":"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"}
Jul 23 11:45:29.791247 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.791230    2583 scope.go:117] "RemoveContainer" containerID="be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"
Jul 23 11:45:29.792303 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.792273    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:29.792381 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.792357    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:29.792428 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:29.792416    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:45:30.797206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:30.797173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerStarted","Data":"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"}
Jul 23 11:45:31.648640 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:31.648607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:31.649117 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:31.649072    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 23 11:45:31.649288 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:31.649263    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 23 11:45:32.648617 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:32.648573    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 23 11:45:32.649234 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:32.649197    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 23 11:45:33.648740 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:33.648692    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 23 11:45:33.649267 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:33.648765    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 23 11:45:34.648771 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:34.648725    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Jul 23 11:45:34.649265 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:34.648793    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Jul 23 11:45:35.651758 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:35.651726    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:35.652218 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:35.651989    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:35.652640 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:35.652626    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:37.976736 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:37.976699    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:37.977215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:37.977005    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" podUID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" containerName="maas-api" containerID="cri-o://45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b" gracePeriod=30
Jul 23 11:45:38.229601 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.229543    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:38.298562 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.298530    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt5hf\" (UniqueName: \"kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf\") pod \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") "
Jul 23 11:45:38.298702 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.298580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls\") pod \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") "
Jul 23 11:45:38.298702 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.298597    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca\") pod \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\" (UID: \"bc1445d4-7888-4f96-91d7-b5d0cc993ed1\") "
Jul 23 11:45:38.298998 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.298971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc1445d4-7888-4f96-91d7-b5d0cc993ed1" (UID: "bc1445d4-7888-4f96-91d7-b5d0cc993ed1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:45:38.300808 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.300781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf" (OuterVolumeSpecName: "kube-api-access-zt5hf") pod "bc1445d4-7888-4f96-91d7-b5d0cc993ed1" (UID: "bc1445d4-7888-4f96-91d7-b5d0cc993ed1"). InnerVolumeSpecName "kube-api-access-zt5hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:45:38.300912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.300882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc1445d4-7888-4f96-91d7-b5d0cc993ed1" (UID: "bc1445d4-7888-4f96-91d7-b5d0cc993ed1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:45:38.399854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.399815    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:38.399854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.399843    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:38.399854 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.399855    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5hf\" (UniqueName: \"kubernetes.io/projected/bc1445d4-7888-4f96-91d7-b5d0cc993ed1-kube-api-access-zt5hf\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:38.831575 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.831544    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" containerID="45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b" exitCode=0
Jul 23 11:45:38.831744 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.831585    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"
Jul 23 11:45:38.831744 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.831629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" event={"ID":"bc1445d4-7888-4f96-91d7-b5d0cc993ed1","Type":"ContainerDied","Data":"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"}
Jul 23 11:45:38.831744 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.831689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw" event={"ID":"bc1445d4-7888-4f96-91d7-b5d0cc993ed1","Type":"ContainerDied","Data":"6cd05ffcc35a3b4e985ea81c9e8bb3aaca68633fe5f072b8927d30bd825439bd"}
Jul 23 11:45:38.831744 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.831719    2583 scope.go:117] "RemoveContainer" containerID="45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"
Jul 23 11:45:38.839912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.839870    2583 scope.go:117] "RemoveContainer" containerID="45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"
Jul 23 11:45:38.840229 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:45:38.840207    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b\": container with ID starting with 45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b not found: ID does not exist" containerID="45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"
Jul 23 11:45:38.840304 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.840239    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b"} err="failed to get container status \"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b\": rpc error: code = NotFound desc = could not find container \"45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b\": container with ID starting with 45dac4f61141867861cc1ae6be63f78def4ad1bd3b040a78670f1d0c277ebe6b not found: ID does not exist"
Jul 23 11:45:38.849461 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.849440    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:38.853391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:38.853371    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3403a9d3-589cccdfdf-jkmqw"]
Jul 23 11:45:40.744541 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:40.744508    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" path="/var/lib/kubelet/pods/bc1445d4-7888-4f96-91d7-b5d0cc993ed1/volumes"
Jul 23 11:45:41.189669 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:41.189624    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:45:49.192176 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:49.192140    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:49.192557 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:49.192434    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" containerID="cri-o://744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333" gracePeriod=30
Jul 23 11:45:50.654829 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:50.654788    2583 patch_prober.go:28] interesting pod/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 23 11:45:50.655225 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:50.654885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 11:45:54.437989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.437964    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:54.542845 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.542845 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542809    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7m97\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.542978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543051 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543043    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy\") pod \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\" (UID: \"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376\") "
Jul 23 11:45:54.543315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:45:54.543315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:45:54.543516 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543401    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.543516 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543419    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.543616 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543574    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:45:54.544142 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.543856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:45:54.544269 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.544031    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:45:54.545421 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.545386    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:45:54.545421 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.545391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:45:54.545581 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.545449    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97" (OuterVolumeSpecName: "kube-api-access-t7m97") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "kube-api-access-t7m97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:45:54.545581 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.545484    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" (UID: "c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:45:54.644169 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644133    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7m97\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-kube-api-access-t7m97\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644169 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644163    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644169 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644178    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644190    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644202    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644214    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.644403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.644224    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:45:54.899753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.899669    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerID="744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333" exitCode=0
Jul 23 11:45:54.899753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.899711    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"
Jul 23 11:45:54.899753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.899741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerDied","Data":"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"}
Jul 23 11:45:54.899988 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.899790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb" event={"ID":"c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376","Type":"ContainerDied","Data":"d87e2c8cd452def699da11e5a9395aba5bc0cb29591caf8aa5f608d731a740cc"}
Jul 23 11:45:54.899988 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.899812    2583 scope.go:117] "RemoveContainer" containerID="744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"
Jul 23 11:45:54.908245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.908226    2583 scope.go:117] "RemoveContainer" containerID="be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"
Jul 23 11:45:54.915549 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.915532    2583 scope.go:117] "RemoveContainer" containerID="744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"
Jul 23 11:45:54.915811 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:45:54.915787    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333\": container with ID starting with 744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333 not found: ID does not exist" containerID="744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"
Jul 23 11:45:54.915902 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.915819    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333"} err="failed to get container status \"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333\": rpc error: code = NotFound desc = could not find container \"744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333\": container with ID starting with 744fb867d8eb16c302ef07170ca77d03f70b05b6109fb3ffaf71e3c1f2080333 not found: ID does not exist"
Jul 23 11:45:54.915902 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.915838    2583 scope.go:117] "RemoveContainer" containerID="be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"
Jul 23 11:45:54.916182 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:45:54.916077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed\": container with ID starting with be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed not found: ID does not exist" containerID="be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"
Jul 23 11:45:54.916283 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.916192    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed"} err="failed to get container status \"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed\": rpc error: code = NotFound desc = could not find container \"be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed\": container with ID starting with be29115a8d6272ef6361cc346204284c807cbc46c3612ae64692251163ac96ed not found: ID does not exist"
Jul 23 11:45:54.918838 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.918813    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:54.922815 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:54.922795    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3403a9d3-openshift-default-7689d749f9-rqbfb"]
Jul 23 11:45:56.744605 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:45:56.744573    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" path="/var/lib/kubelet/pods/c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376/volumes"
Jul 23 11:46:09.011358 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011323    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011599    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011610    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" containerName="maas-api"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011635    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" containerName="maas-api"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011684    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1445d4-7888-4f96-91d7-b5d0cc993ed1" containerName="maas-api"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011693    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:09.011831 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.011698    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:09.014588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.014572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.017524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.017504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d0774c5e-openshift-default-dockercfg-mspwh\""
Jul 23 11:46:09.026469 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.026439    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:46:09.041113 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041304 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041362 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041362 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t29d\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.041443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.041428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t29d\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142793 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.142939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.143005 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.142984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.143045 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.143014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.143261 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.143235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.144739 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.144718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.145232 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.145215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.151798 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.151770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t29d\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.153028 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.153000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token\") pod \"e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") " pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.325989 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.325901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:09.461553 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.461522    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:46:09.463501 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:46:09.463468    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a04192_ee80_47b5_b4cd_902de184c599.slice/crio-c5a6e68c09a5148168d4b9e19b9623c07bc0216b30e92e00f1d0025f264b7e22 WatchSource:0}: Error finding container c5a6e68c09a5148168d4b9e19b9623c07bc0216b30e92e00f1d0025f264b7e22: Status 404 returned error can't find the container with id c5a6e68c09a5148168d4b9e19b9623c07bc0216b30e92e00f1d0025f264b7e22
Jul 23 11:46:09.465664 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.465631    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:09.465732 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.465696    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:09.465732 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.465723    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:09.969041 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.969006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerStarted","Data":"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"}
Jul 23 11:46:09.969215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.969058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerStarted","Data":"c5a6e68c09a5148168d4b9e19b9623c07bc0216b30e92e00f1d0025f264b7e22"}
Jul 23 11:46:09.991042 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:09.990993    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podStartSLOduration=1.990978586 podStartE2EDuration="1.990978586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:09.988979587 +0000 UTC m=+2143.821787845" watchObservedRunningTime="2026-07-23 11:46:09.990978586 +0000 UTC m=+2143.823786819"
Jul 23 11:46:10.327014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:10.326918    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:10.331581 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:10.331555    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:10.972994 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:10.972961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:10.973970 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:10.973948    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:11.940556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.940521    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:46:11.940972 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.940956    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:11.941014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.940976    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ab1667-ecb1-43f8-a17c-ab0ca1bf5376" containerName="istio-proxy"
Jul 23 11:46:11.944294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.944276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:11.946757 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.946728    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d0774c5e-serving-cert\""
Jul 23 11:46:11.946757 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.946747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:46:11.946969 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.946955    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:46:11.947975 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.947958    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:46:11.951059 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.951037    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:46:11.972757 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.972722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdwc\" (UniqueName: \"kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:11.972914 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.972842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:11.972914 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:11.972895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.073948 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.073907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdwc\" (UniqueName: \"kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.074176 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.074146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.074176 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.074169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.074317 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:46:12.074241    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-serving-cert: secret "maas-api-e2e-mt-d0774c5e-serving-cert" not found
Jul 23 11:46:12.074372 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:46:12.074323    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls podName:aabd7dc1-7c36-4649-8f0c-71608f34e0c0 nodeName:}" failed. No retries permitted until 2026-07-23 11:46:12.574307332 +0000 UTC m=+2146.407115547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls") pod "maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" (UID: "aabd7dc1-7c36-4649-8f0c-71608f34e0c0") : secret "maas-api-e2e-mt-d0774c5e-serving-cert" not found
Jul 23 11:46:12.074744 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.074727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.088935 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.088909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdwc\" (UniqueName: \"kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.579316 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.579279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.581966 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.581933    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") pod \"maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.857565 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.857460    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:12.994191 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.994168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:46:12.996628 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:46:12.996602    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabd7dc1_7c36_4649_8f0c_71608f34e0c0.slice/crio-0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0 WatchSource:0}: Error finding container 0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0: Status 404 returned error can't find the container with id 0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0
Jul 23 11:46:12.998197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:12.998170    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:46:13.986909 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:13.986870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" event={"ID":"aabd7dc1-7c36-4649-8f0c-71608f34e0c0","Type":"ContainerStarted","Data":"56d9b8ec9b1154d49564bc441690154a9eb6bf319fb3d83a05e87b6811deec35"}
Jul 23 11:46:13.987148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:13.986940    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:13.987148 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:13.986965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" event={"ID":"aabd7dc1-7c36-4649-8f0c-71608f34e0c0","Type":"ContainerStarted","Data":"0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0"}
Jul 23 11:46:14.005360 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:14.005306    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" podStartSLOduration=2.588116295 podStartE2EDuration="3.005287873s" totalImagesPullingTime="417.171578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:14.003986771 +0000 UTC m=+2147.836795005" watchObservedRunningTime="2026-07-23 11:46:14.005287873 +0000 UTC m=+2147.838096106"
Jul 23 11:46:16.082354 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:16.082312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:46:19.998040 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:19.997962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:46:24.032103 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.032002    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3a04192-ee80-47b5-b4cd-902de184c599" containerID="0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a" exitCode=0
Jul 23 11:46:24.032496 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.032086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerDied","Data":"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"}
Jul 23 11:46:24.032540 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.032503    2583 scope.go:117] "RemoveContainer" containerID="0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"
Jul 23 11:46:24.033587 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.033562    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:24.033656 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.033635    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:24.033690 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:24.033670    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:46:25.044899 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:25.044843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerStarted","Data":"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"}
Jul 23 11:46:25.327676 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:25.327574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:25.327853 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:25.327732    2583 patch_prober.go:28] interesting pod/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 23 11:46:25.327853 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:25.327797    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 23 11:46:25.973195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:25.973141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:26.327377 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:26.327265    2583 patch_prober.go:28] interesting pod/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 23 11:46:26.328378 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:26.327920    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 23 11:46:27.326876 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:27.326835    2583 patch_prober.go:28] interesting pod/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 23 11:46:27.327230 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:27.327196    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 23 11:46:27.680584 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:27.680548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:46:28.326939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:28.326890    2583 patch_prober.go:28] interesting pod/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 23 11:46:28.327239 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:28.327205    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 23 11:46:29.330561 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:29.330529    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:29.331631 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:29.331614    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:46:52.188927 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:46:52.188892    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:47:09.486597 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:09.486553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:47:11.001688 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.001650    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:47:11.002185 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.001861    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" podUID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" containerName="maas-api" containerID="cri-o://56d9b8ec9b1154d49564bc441690154a9eb6bf319fb3d83a05e87b6811deec35" gracePeriod=30
Jul 23 11:47:11.245909 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.245882    2583 generic.go:356] "Generic (PLEG): container finished" podID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" containerID="56d9b8ec9b1154d49564bc441690154a9eb6bf319fb3d83a05e87b6811deec35" exitCode=0
Jul 23 11:47:11.246054 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.245928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" event={"ID":"aabd7dc1-7c36-4649-8f0c-71608f34e0c0","Type":"ContainerDied","Data":"56d9b8ec9b1154d49564bc441690154a9eb6bf319fb3d83a05e87b6811deec35"}
Jul 23 11:47:11.246054 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.245965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8" event={"ID":"aabd7dc1-7c36-4649-8f0c-71608f34e0c0","Type":"ContainerDied","Data":"0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0"}
Jul 23 11:47:11.246054 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.245981    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8a9fddf51cd7cc41cfc5daa5ebe33a11606a325a7e014316d41f1e53ee95f0"
Jul 23 11:47:11.257237 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.257179    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:47:11.311453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.311420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") pod \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") "
Jul 23 11:47:11.311616 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.311526    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kdwc\" (UniqueName: \"kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc\") pod \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") "
Jul 23 11:47:11.311616 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.311602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca\") pod \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\" (UID: \"aabd7dc1-7c36-4649-8f0c-71608f34e0c0\") "
Jul 23 11:47:11.311954 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.311918    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aabd7dc1-7c36-4649-8f0c-71608f34e0c0" (UID: "aabd7dc1-7c36-4649-8f0c-71608f34e0c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:47:11.313826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.313797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc" (OuterVolumeSpecName: "kube-api-access-4kdwc") pod "aabd7dc1-7c36-4649-8f0c-71608f34e0c0" (UID: "aabd7dc1-7c36-4649-8f0c-71608f34e0c0"). InnerVolumeSpecName "kube-api-access-4kdwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:47:11.313933 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.313856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aabd7dc1-7c36-4649-8f0c-71608f34e0c0" (UID: "aabd7dc1-7c36-4649-8f0c-71608f34e0c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:47:11.412672 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.412625    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:47:11.412672 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.412659    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:47:11.412672 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:11.412669    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdwc\" (UniqueName: \"kubernetes.io/projected/aabd7dc1-7c36-4649-8f0c-71608f34e0c0-kube-api-access-4kdwc\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:47:12.250155 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:12.250121    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"
Jul 23 11:47:12.270955 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:12.270911    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:47:12.273059 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:12.273036    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0774c5e-5c9f648c5c-6p9m8"]
Jul 23 11:47:12.744762 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:12.744729    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" path="/var/lib/kubelet/pods/aabd7dc1-7c36-4649-8f0c-71608f34e0c0/volumes"
Jul 23 11:47:20.084028 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:20.083988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:47:26.769449 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:47:26.769415    2583 scope.go:117] "RemoveContainer" containerID="5ca118489152ce4c119b264666ab94ae5032d3c5003cb82b786218c1044e6011"
Jul 23 11:48:05.844622 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:05.844537    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:48:05.845180 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:05.844765    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy" containerID="cri-o://27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1" gracePeriod=30
Jul 23 11:48:09.583070 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:09.583031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:48:11.096563 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.096538    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:48:11.211158 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211125    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211351 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211184    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211351 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211351 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211266    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t29d\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211351 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211351 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211599 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211599 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211599 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket\") pod \"a3a04192-ee80-47b5-b4cd-902de184c599\" (UID: \"a3a04192-ee80-47b5-b4cd-902de184c599\") "
Jul 23 11:48:11.211753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:48:11.211753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:48:11.211852 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211758    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a04192-ee80-47b5-b4cd-902de184c599-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.211852 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211753    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:48:11.211852 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.211778    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.212178 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.212120    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:48:11.212517 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.212498    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:48:11.213908 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.213886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:48:11.214012 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.213930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:48:11.214214 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.214189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d" (OuterVolumeSpecName: "kube-api-access-6t29d") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "kube-api-access-6t29d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:48:11.214214 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.214204    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3a04192-ee80-47b5-b4cd-902de184c599" (UID: "a3a04192-ee80-47b5-b4cd-902de184c599"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:48:11.312289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312242    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312286    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a04192-ee80-47b5-b4cd-902de184c599-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312297    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312289 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312308    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t29d\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-kube-api-access-6t29d\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312615 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312317    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312615 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312326    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a04192-ee80-47b5-b4cd-902de184c599-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.312615 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.312333    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a04192-ee80-47b5-b4cd-902de184c599-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:48:11.505593 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.505508    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3a04192-ee80-47b5-b4cd-902de184c599" containerID="27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1" exitCode=0
Jul 23 11:48:11.505593 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.505561    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"
Jul 23 11:48:11.505766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.505560    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerDied","Data":"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"}
Jul 23 11:48:11.505766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.505691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn" event={"ID":"a3a04192-ee80-47b5-b4cd-902de184c599","Type":"ContainerDied","Data":"c5a6e68c09a5148168d4b9e19b9623c07bc0216b30e92e00f1d0025f264b7e22"}
Jul 23 11:48:11.505766 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.505727    2583 scope.go:117] "RemoveContainer" containerID="27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"
Jul 23 11:48:11.514142 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.514118    2583 scope.go:117] "RemoveContainer" containerID="0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"
Jul 23 11:48:11.522064 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.522041    2583 scope.go:117] "RemoveContainer" containerID="27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"
Jul 23 11:48:11.522328 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:48:11.522307    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1\": container with ID starting with 27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1 not found: ID does not exist" containerID="27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"
Jul 23 11:48:11.522392 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.522337    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1"} err="failed to get container status \"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1\": rpc error: code = NotFound desc = could not find container \"27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1\": container with ID starting with 27bd432b7744134d5a33171abbc529f5c2da343184ba29c6dd404f964c0438f1 not found: ID does not exist"
Jul 23 11:48:11.522392 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.522355    2583 scope.go:117] "RemoveContainer" containerID="0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"
Jul 23 11:48:11.522586 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:48:11.522571    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a\": container with ID starting with 0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a not found: ID does not exist" containerID="0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"
Jul 23 11:48:11.522634 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.522589    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a"} err="failed to get container status \"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a\": rpc error: code = NotFound desc = could not find container \"0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a\": container with ID starting with 0a9fd5ff40b836fceff10597cfdd57246e632d9d79877095e7f5e27838acf94a not found: ID does not exist"
Jul 23 11:48:11.528085 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.528060    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:48:11.529378 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:11.529360    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d0774c5e-openshift-default-67849dd895-z5vkn"]
Jul 23 11:48:12.748542 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:12.748505    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" path="/var/lib/kubelet/pods/a3a04192-ee80-47b5-b4cd-902de184c599/volumes"
Jul 23 11:48:22.091987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:48:22.091948    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:49:01.543042 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543005    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543328    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" containerName="maas-api"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" containerName="maas-api"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543359    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543365    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543381    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543386    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543432    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:01.543524 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.543441    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabd7dc1-7c36-4649-8f0c-71608f34e0c0" containerName="maas-api"
Jul 23 11:49:01.546366 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.546348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.548813 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.548792    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9c6be2-openshift-default-dockercfg-b8l2q\""
Jul 23 11:49:01.559012 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.558986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:49:01.647473 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.647673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.647673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.647673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.647673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgm4\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.647673 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.648152 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.648152 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.648152 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.647847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.748753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.748753 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgm4\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749014 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.748921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749424 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.749315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749485 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.749451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749538 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.749500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749581 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.749530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.749638 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.749613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.751383 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.751362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.751518 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.751500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.757315 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.757282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.757547 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.757526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgm4\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4\") pod \"e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") " pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.856433 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.856335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:01.999077 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:01.998975    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:49:02.001922 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:49:02.001892    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1b26f_8c43_447d_90da_5049834f5883.slice/crio-f1116d540c288c214102c4bfd717b11e1a751ec2adc0d78a72e6aebc05aec742 WatchSource:0}: Error finding container f1116d540c288c214102c4bfd717b11e1a751ec2adc0d78a72e6aebc05aec742: Status 404 returned error can't find the container with id f1116d540c288c214102c4bfd717b11e1a751ec2adc0d78a72e6aebc05aec742
Jul 23 11:49:02.004263 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.004227    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:02.004352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.004303    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:02.004352 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.004332    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:02.727083 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.727043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerStarted","Data":"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"}
Jul 23 11:49:02.727505 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.727130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerStarted","Data":"f1116d540c288c214102c4bfd717b11e1a751ec2adc0d78a72e6aebc05aec742"}
Jul 23 11:49:02.748207 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.748160    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podStartSLOduration=1.748142995 podStartE2EDuration="1.748142995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:49:02.747045988 +0000 UTC m=+2316.579854250" watchObservedRunningTime="2026-07-23 11:49:02.748142995 +0000 UTC m=+2316.580951227"
Jul 23 11:49:02.856748 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:02.856673    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:03.862541 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:03.862509    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:04.721262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.721224    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:49:04.721596 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.721582    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a04192-ee80-47b5-b4cd-902de184c599" containerName="istio-proxy"
Jul 23 11:49:04.724417 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.724392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.727071 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.727050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:49:04.727206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.727108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9c6be2-serving-cert\""
Jul 23 11:49:04.728145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.728124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:49:04.728205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.728160    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:49:04.737222 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.737194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:04.738318 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.738304    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:04.744977 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.744955    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:49:04.875903 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.875864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.876325 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.876016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.876325 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.876187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dp6\" (UniqueName: \"kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.976921 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.976815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.976921 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.976864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.977267 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.976932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dp6\" (UniqueName: \"kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.977706 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.977684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.979592 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.979567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:04.987422 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:04.987393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dp6\" (UniqueName: \"kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6\") pod \"maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:05.035508 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:05.035469    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:05.192536 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:05.192507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:49:05.193897 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:49:05.193870    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80df42f_4930_4619_81e5_4850e0b0a9c0.slice/crio-d854095c4effa826a5cb9e915194dde8a4a4917299c7285e81b83183e952123d WatchSource:0}: Error finding container d854095c4effa826a5cb9e915194dde8a4a4917299c7285e81b83183e952123d: Status 404 returned error can't find the container with id d854095c4effa826a5cb9e915194dde8a4a4917299c7285e81b83183e952123d
Jul 23 11:49:05.742498 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:05.742464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" event={"ID":"f80df42f-4930-4619-81e5-4850e0b0a9c0","Type":"ContainerStarted","Data":"d854095c4effa826a5cb9e915194dde8a4a4917299c7285e81b83183e952123d"}
Jul 23 11:49:06.748086 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:06.748046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" event={"ID":"f80df42f-4930-4619-81e5-4850e0b0a9c0","Type":"ContainerStarted","Data":"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"}
Jul 23 11:49:06.748684 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:06.748253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:06.765197 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:06.765150    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" podStartSLOduration=2.054196914 podStartE2EDuration="2.765137247s" totalImagesPullingTime="710.940333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:49:06.764607045 +0000 UTC m=+2320.597415282" watchObservedRunningTime="2026-07-23 11:49:06.765137247 +0000 UTC m=+2320.597945480"
Jul 23 11:49:10.577682 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:10.577642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:49:12.758665 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:12.758639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:49:12.909769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:12.909733    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:49:12.913692 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:12.913664    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:12.916175 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:12.916150    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4c7f56-serving-cert\""
Jul 23 11:49:12.924975 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:12.924946    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:49:13.051826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.051587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.051826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.051644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kk4s\" (UniqueName: \"kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.051826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.051666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.152278 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.152234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kk4s\" (UniqueName: \"kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.152278 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.152282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.152526 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.152373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.152526 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:49:13.152419    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-serving-cert: secret "maas-api-e2e-inf-b-4c7f56-serving-cert" not found
Jul 23 11:49:13.152526 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:49:13.152495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls podName:29467405-6f1a-4683-8213-21afba2d6997 nodeName:}" failed. No retries permitted until 2026-07-23 11:49:13.652475228 +0000 UTC m=+2327.485283456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls") pod "maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" (UID: "29467405-6f1a-4683-8213-21afba2d6997") : secret "maas-api-e2e-inf-b-4c7f56-serving-cert" not found
Jul 23 11:49:13.153256 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.153234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.163724 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.163700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kk4s\" (UniqueName: \"kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.657109 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.657066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.659826 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.659792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.834390 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.834348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:13.999827 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:13.999787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:49:14.002381 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:49:14.002353    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29467405_6f1a_4683_8213_21afba2d6997.slice/crio-9e0b65f8a1b83ca901d90b2142a64bbdf7f06c67b5c1a0ca75fa6204a0fec5d5 WatchSource:0}: Error finding container 9e0b65f8a1b83ca901d90b2142a64bbdf7f06c67b5c1a0ca75fa6204a0fec5d5: Status 404 returned error can't find the container with id 9e0b65f8a1b83ca901d90b2142a64bbdf7f06c67b5c1a0ca75fa6204a0fec5d5
Jul 23 11:49:14.786857 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:14.786815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" event={"ID":"29467405-6f1a-4683-8213-21afba2d6997","Type":"ContainerStarted","Data":"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"}
Jul 23 11:49:14.787072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:14.786908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" event={"ID":"29467405-6f1a-4683-8213-21afba2d6997","Type":"ContainerStarted","Data":"9e0b65f8a1b83ca901d90b2142a64bbdf7f06c67b5c1a0ca75fa6204a0fec5d5"}
Jul 23 11:49:14.787072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:14.786960    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:14.806403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:14.806344    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" podStartSLOduration=2.431344097 podStartE2EDuration="2.806324145s" totalImagesPullingTime="374.980048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:49:14.806139973 +0000 UTC m=+2328.638948206" watchObservedRunningTime="2026-07-23 11:49:14.806324145 +0000 UTC m=+2328.639132380"
Jul 23 11:49:18.807398 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.807319    2583 generic.go:356] "Generic (PLEG): container finished" podID="82d1b26f-8c43-447d-90da-5049834f5883" containerID="d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454" exitCode=0
Jul 23 11:49:18.807805 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.807393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerDied","Data":"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"}
Jul 23 11:49:18.807805 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.807764    2583 scope.go:117] "RemoveContainer" containerID="d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"
Jul 23 11:49:18.808758 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.808729    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:18.808859 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.808814    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:18.808921 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:18.808861    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:49:19.737879 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:19.737823    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:19.814049 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:19.814006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerStarted","Data":"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"}
Jul 23 11:49:19.856525 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:19.856492    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:19.857619 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:19.857591    2583 patch_prober.go:28] interesting pod/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 23 11:49:19.857767 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:19.857649    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 23 11:49:20.797479 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:20.797449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:49:20.857587 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:20.857558    2583 patch_prober.go:28] interesting pod/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 23 11:49:20.858052 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:20.857723    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 23 11:49:21.857613 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:21.857567    2583 patch_prober.go:28] interesting pod/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 23 11:49:21.858013 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:21.857629    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 23 11:49:22.857016 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:22.856980    2583 patch_prober.go:28] interesting pod/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 23 11:49:22.857229 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:22.857039    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 23 11:49:22.883974 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:22.883936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:49:23.872431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:23.872399    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:23.872825 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:23.872797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:23.873482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:23.873465    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:49:26.798243 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:26.798212    2583 scope.go:117] "RemoveContainer" containerID="85c2bbd2c8c0952c211ed23b3233a0b615d976fdeb0c365f315ecc14aa613059"
Jul 23 11:49:37.982606 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:37.982560    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:49:51.684779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:49:51.684747    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:50:24.411313 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:50:24.411275    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:51:17.381262 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:17.381227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:51:23.880774 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:23.880739    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:51:23.881219 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:23.880934    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" podUID="f80df42f-4930-4619-81e5-4850e0b0a9c0" containerName="maas-api" containerID="cri-o://93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0" gracePeriod=30
Jul 23 11:51:24.136179 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.136126    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:51:24.304578 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.304546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca\") pod \"f80df42f-4930-4619-81e5-4850e0b0a9c0\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") "
Jul 23 11:51:24.304771 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.304598    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls\") pod \"f80df42f-4930-4619-81e5-4850e0b0a9c0\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") "
Jul 23 11:51:24.304771 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.304627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9dp6\" (UniqueName: \"kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6\") pod \"f80df42f-4930-4619-81e5-4850e0b0a9c0\" (UID: \"f80df42f-4930-4619-81e5-4850e0b0a9c0\") "
Jul 23 11:51:24.304904 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.304877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f80df42f-4930-4619-81e5-4850e0b0a9c0" (UID: "f80df42f-4930-4619-81e5-4850e0b0a9c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:51:24.306816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.306785    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f80df42f-4930-4619-81e5-4850e0b0a9c0" (UID: "f80df42f-4930-4619-81e5-4850e0b0a9c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:51:24.306816 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.306795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6" (OuterVolumeSpecName: "kube-api-access-c9dp6") pod "f80df42f-4930-4619-81e5-4850e0b0a9c0" (UID: "f80df42f-4930-4619-81e5-4850e0b0a9c0"). InnerVolumeSpecName "kube-api-access-c9dp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:51:24.344689 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.344655    2583 generic.go:356] "Generic (PLEG): container finished" podID="f80df42f-4930-4619-81e5-4850e0b0a9c0" containerID="93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0" exitCode=0
Jul 23 11:51:24.344836 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.344700    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"
Jul 23 11:51:24.344836 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.344746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" event={"ID":"f80df42f-4930-4619-81e5-4850e0b0a9c0","Type":"ContainerDied","Data":"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"}
Jul 23 11:51:24.344836 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.344806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w" event={"ID":"f80df42f-4930-4619-81e5-4850e0b0a9c0","Type":"ContainerDied","Data":"d854095c4effa826a5cb9e915194dde8a4a4917299c7285e81b83183e952123d"}
Jul 23 11:51:24.344836 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.344835    2583 scope.go:117] "RemoveContainer" containerID="93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"
Jul 23 11:51:24.353883 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.353866    2583 scope.go:117] "RemoveContainer" containerID="93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"
Jul 23 11:51:24.354166 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:51:24.354143    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0\": container with ID starting with 93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0 not found: ID does not exist" containerID="93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"
Jul 23 11:51:24.354219 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.354180    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0"} err="failed to get container status \"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0\": rpc error: code = NotFound desc = could not find container \"93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0\": container with ID starting with 93659e949d83a714eac2ec40fb5844c693d94d9f90a383f8748abf30c09c46b0 not found: ID does not exist"
Jul 23 11:51:24.365445 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.365417    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:51:24.370761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.370743    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c6be2-c7849dd47-sl48w"]
Jul 23 11:51:24.406167 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.406113    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f80df42f-4930-4619-81e5-4850e0b0a9c0-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:24.406167 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.406134    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f80df42f-4930-4619-81e5-4850e0b0a9c0-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:24.406167 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.406145    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9dp6\" (UniqueName: \"kubernetes.io/projected/f80df42f-4930-4619-81e5-4850e0b0a9c0-kube-api-access-c9dp6\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:24.744939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:24.744906    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f80df42f-4930-4619-81e5-4850e0b0a9c0" path="/var/lib/kubelet/pods/f80df42f-4930-4619-81e5-4850e0b0a9c0/volumes"
Jul 23 11:51:32.582960 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:32.582912    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:51:34.843074 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:34.843039    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:51:34.843520 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:34.843297    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" containerID="cri-o://67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026" gracePeriod=30
Jul 23 11:51:38.874623 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:38.874583    2583 patch_prober.go:28] interesting pod/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 11:51:38.875024 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:38.874657    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 11:51:40.105187 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.105119    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:51:40.126438 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126483    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126522    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126572    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzgm4\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126823 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126604    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.126823 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.126650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.127083 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127448    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs\") pod \"82d1b26f-8c43-447d-90da-5049834f5883\" (UID: \"82d1b26f-8c43-447d-90da-5049834f5883\") "
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127777    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127801    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d1b26f-8c43-447d-90da-5049834f5883-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.127907    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data" (OuterVolumeSpecName: "istio-data") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.128016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:51:40.129220 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.128138    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:51:40.130194 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.130157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:51:40.130314 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.130260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:51:40.130664 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.130626    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4" (OuterVolumeSpecName: "kube-api-access-xzgm4") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "kube-api-access-xzgm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:51:40.130841 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.130816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token" (OuterVolumeSpecName: "istio-token") pod "82d1b26f-8c43-447d-90da-5049834f5883" (UID: "82d1b26f-8c43-447d-90da-5049834f5883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:51:40.228510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228464    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzgm4\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-kube-api-access-xzgm4\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228503    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228513    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d1b26f-8c43-447d-90da-5049834f5883-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228522    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228530    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228839 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228538    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d1b26f-8c43-447d-90da-5049834f5883-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.228839 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.228545    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d1b26f-8c43-447d-90da-5049834f5883-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:40.424798 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.424703    2583 generic.go:356] "Generic (PLEG): container finished" podID="82d1b26f-8c43-447d-90da-5049834f5883" containerID="67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026" exitCode=0
Jul 23 11:51:40.424798 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.424744    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"
Jul 23 11:51:40.424798 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.424784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerDied","Data":"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"}
Jul 23 11:51:40.425061 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.424840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr" event={"ID":"82d1b26f-8c43-447d-90da-5049834f5883","Type":"ContainerDied","Data":"f1116d540c288c214102c4bfd717b11e1a751ec2adc0d78a72e6aebc05aec742"}
Jul 23 11:51:40.425061 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.424862    2583 scope.go:117] "RemoveContainer" containerID="67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"
Jul 23 11:51:40.439405 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.434502    2583 scope.go:117] "RemoveContainer" containerID="d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"
Jul 23 11:51:40.447933 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.447912    2583 scope.go:117] "RemoveContainer" containerID="67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"
Jul 23 11:51:40.448269 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:51:40.448243    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026\": container with ID starting with 67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026 not found: ID does not exist" containerID="67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"
Jul 23 11:51:40.448336 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.448281    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026"} err="failed to get container status \"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026\": rpc error: code = NotFound desc = could not find container \"67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026\": container with ID starting with 67ec86ce622580e3c81fd54ac8e9107151bfaeece7b1a7a18ef237366c23c026 not found: ID does not exist"
Jul 23 11:51:40.448336 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.448306    2583 scope.go:117] "RemoveContainer" containerID="d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"
Jul 23 11:51:40.448596 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:51:40.448576    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454\": container with ID starting with d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454 not found: ID does not exist" containerID="d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"
Jul 23 11:51:40.448662 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.448606    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454"} err="failed to get container status \"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454\": rpc error: code = NotFound desc = could not find container \"d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454\": container with ID starting with d22bd1742721b17b69a0cf9f770b9a37110a707f2c74cfd62d0598e55abdf454 not found: ID does not exist"
Jul 23 11:51:40.449037 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.449017    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:51:40.453602 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.453579    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9c6be2-openshift-default-dc84d9955-6jswr"]
Jul 23 11:51:40.745164 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:40.745131    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d1b26f-8c43-447d-90da-5049834f5883" path="/var/lib/kubelet/pods/82d1b26f-8c43-447d-90da-5049834f5883/volumes"
Jul 23 11:51:45.591388 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:45.591354    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:51:45.591817 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:45.591572    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" podUID="29467405-6f1a-4683-8213-21afba2d6997" containerName="maas-api" containerID="cri-o://aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f" gracePeriod=30
Jul 23 11:51:45.792890 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:45.792855    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" podUID="29467405-6f1a-4683-8213-21afba2d6997" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused"
Jul 23 11:51:46.029242 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.029215    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:51:46.085322 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.085291    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") pod \"29467405-6f1a-4683-8213-21afba2d6997\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") "
Jul 23 11:51:46.085443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.085346    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kk4s\" (UniqueName: \"kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s\") pod \"29467405-6f1a-4683-8213-21afba2d6997\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") "
Jul 23 11:51:46.085443 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.085378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca\") pod \"29467405-6f1a-4683-8213-21afba2d6997\" (UID: \"29467405-6f1a-4683-8213-21afba2d6997\") "
Jul 23 11:51:46.085769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.085746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29467405-6f1a-4683-8213-21afba2d6997" (UID: "29467405-6f1a-4683-8213-21afba2d6997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:51:46.087612 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.087581    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s" (OuterVolumeSpecName: "kube-api-access-4kk4s") pod "29467405-6f1a-4683-8213-21afba2d6997" (UID: "29467405-6f1a-4683-8213-21afba2d6997"). InnerVolumeSpecName "kube-api-access-4kk4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:51:46.087709 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.087659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29467405-6f1a-4683-8213-21afba2d6997" (UID: "29467405-6f1a-4683-8213-21afba2d6997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:51:46.185935 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.185906    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29467405-6f1a-4683-8213-21afba2d6997-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:46.185935 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.185933    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kk4s\" (UniqueName: \"kubernetes.io/projected/29467405-6f1a-4683-8213-21afba2d6997-kube-api-access-4kk4s\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:46.186179 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.185945    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29467405-6f1a-4683-8213-21afba2d6997-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:51:46.456370 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.456263    2583 generic.go:356] "Generic (PLEG): container finished" podID="29467405-6f1a-4683-8213-21afba2d6997" containerID="aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f" exitCode=0
Jul 23 11:51:46.456370 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.456309    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"
Jul 23 11:51:46.456370 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.456338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" event={"ID":"29467405-6f1a-4683-8213-21afba2d6997","Type":"ContainerDied","Data":"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"}
Jul 23 11:51:46.456614 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.456388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc" event={"ID":"29467405-6f1a-4683-8213-21afba2d6997","Type":"ContainerDied","Data":"9e0b65f8a1b83ca901d90b2142a64bbdf7f06c67b5c1a0ca75fa6204a0fec5d5"}
Jul 23 11:51:46.456614 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.456411    2583 scope.go:117] "RemoveContainer" containerID="aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"
Jul 23 11:51:46.465396 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.465379    2583 scope.go:117] "RemoveContainer" containerID="aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"
Jul 23 11:51:46.465660 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:51:46.465639    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f\": container with ID starting with aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f not found: ID does not exist" containerID="aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"
Jul 23 11:51:46.465714 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.465667    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f"} err="failed to get container status \"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f\": rpc error: code = NotFound desc = could not find container \"aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f\": container with ID starting with aaef0c5bf79a7ad5544a58d4abe097a79343353eb9c6f1d24383d7dfc554b92f not found: ID does not exist"
Jul 23 11:51:46.478354 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.478332    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:51:46.483561 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.483539    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c7f56-765c9fc7b4-4hkcc"]
Jul 23 11:51:46.744160 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:46.744039    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29467405-6f1a-4683-8213-21afba2d6997" path="/var/lib/kubelet/pods/29467405-6f1a-4683-8213-21afba2d6997/volumes"
Jul 23 11:51:48.480987 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:48.480952    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:51:57.571036 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571004    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571372    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571384    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571392    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29467405-6f1a-4683-8213-21afba2d6997" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571398    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="29467405-6f1a-4683-8213-21afba2d6997" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571418    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f80df42f-4930-4619-81e5-4850e0b0a9c0" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f80df42f-4930-4619-81e5-4850e0b0a9c0" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571473    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f80df42f-4930-4619-81e5-4850e0b0a9c0" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571480    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="29467405-6f1a-4683-8213-21afba2d6997" containerName="maas-api"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571486    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:51:57.571530 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.571491    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:51:57.575618 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.575602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.578677 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.578661    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-02a4cf-openshift-default-dockercfg-j9qzm\""
Jul 23 11:51:57.598283 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.598254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:51:57.685474 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685474 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv9h\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.685942 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.685734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786551 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786735 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786735 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786735 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv9h\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786899 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.786899 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787024 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787024 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.786962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787024 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787285 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787369 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787439 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787485 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.787568 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.787544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.789008 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.788991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.789246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.789228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.794697 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.794672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.794985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.794965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv9h\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h\") pod \"e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") " pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:57.885334 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:57.885236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:58.024005 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.023977    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:51:58.025953 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:51:58.025915    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71273b2_6bfe_4165_862c_e846cdede6c7.slice/crio-31a2c87189f528b2ef2c69866179e5b84f231627f5825fd40c477de80e3b86c7 WatchSource:0}: Error finding container 31a2c87189f528b2ef2c69866179e5b84f231627f5825fd40c477de80e3b86c7: Status 404 returned error can't find the container with id 31a2c87189f528b2ef2c69866179e5b84f231627f5825fd40c477de80e3b86c7
Jul 23 11:51:58.028301 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.028263    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:51:58.028391 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.028345    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:51:58.028432 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.028389    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:51:58.515969 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.515932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerStarted","Data":"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"}
Jul 23 11:51:58.516205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.515989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerStarted","Data":"31a2c87189f528b2ef2c69866179e5b84f231627f5825fd40c477de80e3b86c7"}
Jul 23 11:51:58.540912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.540853    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podStartSLOduration=1.540836085 podStartE2EDuration="1.540836085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:51:58.540249516 +0000 UTC m=+2492.373057749" watchObservedRunningTime="2026-07-23 11:51:58.540836085 +0000 UTC m=+2492.373644300"
Jul 23 11:51:58.885761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:58.885662    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:51:59.890384 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:51:59.890352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:00.524649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.524610    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:00.525772 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.525749    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:00.668912 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.668877    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:00.669190 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.669177    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:52:00.669190 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.669191    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1b26f-8c43-447d-90da-5049834f5883" containerName="istio-proxy"
Jul 23 11:52:00.672212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.672187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.674954 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.674922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-02a4cf-serving-cert\""
Jul 23 11:52:00.675112 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.674932    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:52:00.675112 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.675018    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:52:00.675437 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.675423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:52:00.683394 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.683372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:00.813146 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.813041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.813146 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.813082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmxn\" (UniqueName: \"kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.813350 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.813265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.914616 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.914587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.915037 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.914629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.915037 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.914661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxn\" (UniqueName: \"kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.915364 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.915345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.917232 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.917214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.924406 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.924379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmxn\" (UniqueName: \"kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn\") pod \"maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:00.983354 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:00.983321    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:01.128624 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:01.128593    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:01.129827 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:52:01.129799    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb272f586_bd3f_4351_a20c_f44c7d9485e4.slice/crio-960d67ad137f0331291e61241452e8cc0cd1e253df90061d54c8ec4776b5d6e0 WatchSource:0}: Error finding container 960d67ad137f0331291e61241452e8cc0cd1e253df90061d54c8ec4776b5d6e0: Status 404 returned error can't find the container with id 960d67ad137f0331291e61241452e8cc0cd1e253df90061d54c8ec4776b5d6e0
Jul 23 11:52:01.131210 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:01.131192    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:52:01.529428 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:01.529389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" event={"ID":"b272f586-bd3f-4351-a20c-f44c7d9485e4","Type":"ContainerStarted","Data":"960d67ad137f0331291e61241452e8cc0cd1e253df90061d54c8ec4776b5d6e0"}
Jul 23 11:52:02.534784 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:02.534739    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" event={"ID":"b272f586-bd3f-4351-a20c-f44c7d9485e4","Type":"ContainerStarted","Data":"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"}
Jul 23 11:52:02.535195 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:02.534824    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:02.555949 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:02.555871    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" podStartSLOduration=2.087718935 podStartE2EDuration="2.55585683s" totalImagesPullingTime="468.137895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:52:02.554867816 +0000 UTC m=+2496.387676085" watchObservedRunningTime="2026-07-23 11:52:02.55585683 +0000 UTC m=+2496.388665063"
Jul 23 11:52:08.493202 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.493169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:52:08.546490 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.546459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:08.855814 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.855733    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:52:08.859042 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.859020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:08.861754 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.861722    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a9439f-serving-cert\""
Jul 23 11:52:08.870765 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.870724    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:52:08.981379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.981331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5lz\" (UniqueName: \"kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:08.981576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.981388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:08.981576 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:08.981432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.081985 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.081956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5lz\" (UniqueName: \"kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.082206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.081991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.082206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.082017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.082206 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:52:09.082154    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-serving-cert: secret "maas-api-e2e-api-b-a9439f-serving-cert" not found
Jul 23 11:52:09.082390 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:52:09.082211    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls podName:ca1c117a-35c2-4f5e-b9e9-000ca3fa3567 nodeName:}" failed. No retries permitted until 2026-07-23 11:52:09.58219584 +0000 UTC m=+2503.415004053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls") pod "maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" (UID: "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567") : secret "maas-api-e2e-api-b-a9439f-serving-cert" not found
Jul 23 11:52:09.082762 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.082736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.090566 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.090545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5lz\" (UniqueName: \"kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.586476 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.586437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.589135 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.589111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") pod \"maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.770857 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.770818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:09.947341 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:09.947241    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:52:09.957236 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:52:09.957204    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1c117a_35c2_4f5e_b9e9_000ca3fa3567.slice/crio-f057f8363a5d6540e439de14ab49fdd6321c77f8f21fa4a3cb537bcc8d498eee WatchSource:0}: Error finding container f057f8363a5d6540e439de14ab49fdd6321c77f8f21fa4a3cb537bcc8d498eee: Status 404 returned error can't find the container with id f057f8363a5d6540e439de14ab49fdd6321c77f8f21fa4a3cb537bcc8d498eee
Jul 23 11:52:10.571008 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:10.570970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" event={"ID":"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567","Type":"ContainerStarted","Data":"f057f8363a5d6540e439de14ab49fdd6321c77f8f21fa4a3cb537bcc8d498eee"}
Jul 23 11:52:11.576605 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:11.576570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" event={"ID":"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567","Type":"ContainerStarted","Data":"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"}
Jul 23 11:52:11.577010 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:11.576658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:11.598005 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:11.597959    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" podStartSLOduration=2.889415504 podStartE2EDuration="3.597946852s" totalImagesPullingTime="708.531348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:52:11.596538969 +0000 UTC m=+2505.429347204" watchObservedRunningTime="2026-07-23 11:52:11.597946852 +0000 UTC m=+2505.430755085"
Jul 23 11:52:16.567242 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:16.567201    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 23 11:52:16.576864 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:16.567276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 11:52:17.592444 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.592371    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:52:17.614145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.614116    2583 generic.go:356] "Generic (PLEG): container finished" podID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerID="33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947" exitCode=0
Jul 23 11:52:17.614285 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.614190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerDied","Data":"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"}
Jul 23 11:52:17.614572 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.614557    2583 scope.go:117] "RemoveContainer" containerID="33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"
Jul 23 11:52:17.615523 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.615496    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:52:17.615611 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.615556    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:52:17.615611 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:17.615588    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:52:18.620779 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:18.620743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerStarted","Data":"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"}
Jul 23 11:52:18.885526 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:18.885423    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:18.885714 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:18.885685    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 23 11:52:18.885765 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:18.885743    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 23 11:52:19.886205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:19.886167    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 23 11:52:19.886607 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:19.886224    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 23 11:52:20.886217 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:20.886181    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 23 11:52:20.886610 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:20.886239    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 23 11:52:21.886040 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:21.886007    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 23 11:52:21.886202 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:21.886058    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 23 11:52:22.889480 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:22.889454    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:22.889859 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:22.889745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:22.890482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:22.890457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:23.894420 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:23.894378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:52:26.831488 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:26.831454    2583 scope.go:117] "RemoveContainer" containerID="56d9b8ec9b1154d49564bc441690154a9eb6bf319fb3d83a05e87b6811deec35"
Jul 23 11:52:36.319750 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.319717    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:36.320142 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.319966    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" podUID="b272f586-bd3f-4351-a20c-f44c7d9485e4" containerName="maas-api" containerID="cri-o://9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc" gracePeriod=30
Jul 23 11:52:36.571862 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.571799    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:36.633082 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.633045    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca\") pod \"b272f586-bd3f-4351-a20c-f44c7d9485e4\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") "
Jul 23 11:52:36.633279 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.633130    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vmxn\" (UniqueName: \"kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn\") pod \"b272f586-bd3f-4351-a20c-f44c7d9485e4\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") "
Jul 23 11:52:36.633279 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.633153    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls\") pod \"b272f586-bd3f-4351-a20c-f44c7d9485e4\" (UID: \"b272f586-bd3f-4351-a20c-f44c7d9485e4\") "
Jul 23 11:52:36.633461 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.633431    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b272f586-bd3f-4351-a20c-f44c7d9485e4" (UID: "b272f586-bd3f-4351-a20c-f44c7d9485e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:52:36.635295 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.635267    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn" (OuterVolumeSpecName: "kube-api-access-7vmxn") pod "b272f586-bd3f-4351-a20c-f44c7d9485e4" (UID: "b272f586-bd3f-4351-a20c-f44c7d9485e4"). InnerVolumeSpecName "kube-api-access-7vmxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:52:36.635403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.635384    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b272f586-bd3f-4351-a20c-f44c7d9485e4" (UID: "b272f586-bd3f-4351-a20c-f44c7d9485e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:52:36.702580 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.702553    2583 generic.go:356] "Generic (PLEG): container finished" podID="b272f586-bd3f-4351-a20c-f44c7d9485e4" containerID="9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc" exitCode=0
Jul 23 11:52:36.702708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.702597    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"
Jul 23 11:52:36.702708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.702617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" event={"ID":"b272f586-bd3f-4351-a20c-f44c7d9485e4","Type":"ContainerDied","Data":"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"}
Jul 23 11:52:36.702708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.702658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6" event={"ID":"b272f586-bd3f-4351-a20c-f44c7d9485e4","Type":"ContainerDied","Data":"960d67ad137f0331291e61241452e8cc0cd1e253df90061d54c8ec4776b5d6e0"}
Jul 23 11:52:36.702708 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.702679    2583 scope.go:117] "RemoveContainer" containerID="9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"
Jul 23 11:52:36.710828 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.710811    2583 scope.go:117] "RemoveContainer" containerID="9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"
Jul 23 11:52:36.711140 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:52:36.711108    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc\": container with ID starting with 9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc not found: ID does not exist" containerID="9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"
Jul 23 11:52:36.711233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.711147    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc"} err="failed to get container status \"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc\": rpc error: code = NotFound desc = could not find container \"9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc\": container with ID starting with 9d881cc2a2d3d695d354b38921876f8eaee65ffb1d9d0eb5338d3abe129c7cfc not found: ID does not exist"
Jul 23 11:52:36.725629 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.725604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:36.730429 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.730410    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-02a4cf-58f9c997cf-prtg6"]
Jul 23 11:52:36.734294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.734274    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b272f586-bd3f-4351-a20c-f44c7d9485e4-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:36.734294 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.734295    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmxn\" (UniqueName: \"kubernetes.io/projected/b272f586-bd3f-4351-a20c-f44c7d9485e4-kube-api-access-7vmxn\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:36.734415 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.734307    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b272f586-bd3f-4351-a20c-f44c7d9485e4-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:36.744259 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:36.744239    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b272f586-bd3f-4351-a20c-f44c7d9485e4" path="/var/lib/kubelet/pods/b272f586-bd3f-4351-a20c-f44c7d9485e4/volumes"
Jul 23 11:52:39.787510 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:39.787476    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:52:48.290401 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:48.290361    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:52:48.290904 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:48.290578    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" containerID="cri-o://20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476" gracePeriod=30
Jul 23 11:52:52.892328 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:52.892287    2583 patch_prober.go:28] interesting pod/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 11:52:52.892693 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:52.892347    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 11:52:53.542663 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.542642    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:53.681466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681633 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681633 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681633 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681633 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jv9h\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681633 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681619    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:52:53.681888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681767    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.681888 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:52:53.682086 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy\") pod \"e71273b2-6bfe-4165-862c-e846cdede6c7\" (UID: \"e71273b2-6bfe-4165-862c-e846cdede6c7\") "
Jul 23 11:52:53.682086 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.681992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:52:53.682215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.682151    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.682215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.682167    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.682215 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.682175    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71273b2-6bfe-4165-862c-e846cdede6c7-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.682534 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.682241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:52:53.682534 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.682401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:52:53.683996 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.683970    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:52:53.683996 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.683979    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h" (OuterVolumeSpecName: "kube-api-access-8jv9h") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "kube-api-access-8jv9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:52:53.684482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.684453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 11:52:53.684482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.684467    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e71273b2-6bfe-4165-862c-e846cdede6c7" (UID: "e71273b2-6bfe-4165-862c-e846cdede6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:52:53.776314 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.776277    2583 generic.go:356] "Generic (PLEG): container finished" podID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerID="20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476" exitCode=0
Jul 23 11:52:53.776466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.776317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerDied","Data":"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"}
Jul 23 11:52:53.776466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.776329    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"
Jul 23 11:52:53.776466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.776369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw" event={"ID":"e71273b2-6bfe-4165-862c-e846cdede6c7","Type":"ContainerDied","Data":"31a2c87189f528b2ef2c69866179e5b84f231627f5825fd40c477de80e3b86c7"}
Jul 23 11:52:53.776466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.776391    2583 scope.go:117] "RemoveContainer" containerID="20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"
Jul 23 11:52:53.783291 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783265    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.783291 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783291    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jv9h\" (UniqueName: \"kubernetes.io/projected/e71273b2-6bfe-4165-862c-e846cdede6c7-kube-api-access-8jv9h\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.783482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783306    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.783482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783322    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.783482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783335    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.783482 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.783347    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71273b2-6bfe-4165-862c-e846cdede6c7-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:52:53.785334 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.785317    2583 scope.go:117] "RemoveContainer" containerID="33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"
Jul 23 11:52:53.792686 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.792669    2583 scope.go:117] "RemoveContainer" containerID="20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"
Jul 23 11:52:53.792913 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:52:53.792898    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476\": container with ID starting with 20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476 not found: ID does not exist" containerID="20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"
Jul 23 11:52:53.792969 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.792921    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476"} err="failed to get container status \"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476\": rpc error: code = NotFound desc = could not find container \"20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476\": container with ID starting with 20dda2a41ceaff031ce15efb3e02851612b5faf486032b01b1b1026e0326c476 not found: ID does not exist"
Jul 23 11:52:53.792969 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.792936    2583 scope.go:117] "RemoveContainer" containerID="33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"
Jul 23 11:52:53.793197 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:52:53.793180    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947\": container with ID starting with 33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947 not found: ID does not exist" containerID="33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"
Jul 23 11:52:53.793247 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.793204    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947"} err="failed to get container status \"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947\": rpc error: code = NotFound desc = could not find container \"33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947\": container with ID starting with 33eb83ffb5d518dccfc546c1438b5915e885c36084794fc98cff15f31ccbe947 not found: ID does not exist"
Jul 23 11:52:53.798600 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.798578    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:52:53.803705 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:53.803684    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-02a4cf-openshift-default-7f869fb5f5-w9tgw"]
Jul 23 11:52:54.745529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:52:54.745494    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" path="/var/lib/kubelet/pods/e71273b2-6bfe-4165-862c-e846cdede6c7/volumes"
Jul 23 11:53:04.550429 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.550395    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:53:04.550911 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.550643    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" podUID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" containerName="maas-api" containerID="cri-o://f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5" gracePeriod=30
Jul 23 11:53:04.802390 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.802334    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:53:04.824072 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.824041    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" containerID="f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5" exitCode=0
Jul 23 11:53:04.824246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.824077    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"
Jul 23 11:53:04.824246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.824132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" event={"ID":"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567","Type":"ContainerDied","Data":"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"}
Jul 23 11:53:04.824246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.824204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb" event={"ID":"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567","Type":"ContainerDied","Data":"f057f8363a5d6540e439de14ab49fdd6321c77f8f21fa4a3cb537bcc8d498eee"}
Jul 23 11:53:04.824246 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.824232    2583 scope.go:117] "RemoveContainer" containerID="f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"
Jul 23 11:53:04.832701 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.832682    2583 scope.go:117] "RemoveContainer" containerID="f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"
Jul 23 11:53:04.832932 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:53:04.832916    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5\": container with ID starting with f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5 not found: ID does not exist" containerID="f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"
Jul 23 11:53:04.832995 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.832939    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5"} err="failed to get container status \"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5\": rpc error: code = NotFound desc = could not find container \"f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5\": container with ID starting with f9820aaaf9dd7231b1d04c4b3eebd2c2cbe28526464de096931b36501bf30bd5 not found: ID does not exist"
Jul 23 11:53:04.872331 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.872308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk5lz\" (UniqueName: \"kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz\") pod \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") "
Jul 23 11:53:04.872453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.872348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") pod \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") "
Jul 23 11:53:04.872453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.872402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca\") pod \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\" (UID: \"ca1c117a-35c2-4f5e-b9e9-000ca3fa3567\") "
Jul 23 11:53:04.872803 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.872780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" (UID: "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:53:04.874573 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.874546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz" (OuterVolumeSpecName: "kube-api-access-hk5lz") pod "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" (UID: "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567"). InnerVolumeSpecName "kube-api-access-hk5lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:53:04.874573 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.874556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" (UID: "ca1c117a-35c2-4f5e-b9e9-000ca3fa3567"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:53:04.973864 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.973830    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk5lz\" (UniqueName: \"kubernetes.io/projected/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-kube-api-access-hk5lz\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:53:04.973864 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.973858    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:53:04.973864 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:04.973868    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:53:05.153553 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:05.153519    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:53:05.159206 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:05.159180    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9439f-6d9fdbb4fd-s5stb"]
Jul 23 11:53:06.744388 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:06.744359    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" path="/var/lib/kubelet/pods/ca1c117a-35c2-4f5e-b9e9-000ca3fa3567/volumes"
Jul 23 11:53:09.618799 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:09.618760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:53:16.183769 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.183734    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184204    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" containerName="maas-api"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184221    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" containerName="maas-api"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184236    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b272f586-bd3f-4351-a20c-f44c7d9485e4" containerName="maas-api"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184244    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b272f586-bd3f-4351-a20c-f44c7d9485e4" containerName="maas-api"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184256    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184265    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184280    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:16.184335 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184288    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:16.184761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184345    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:16.184761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184356    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1c117a-35c2-4f5e-b9e9-000ca3fa3567" containerName="maas-api"
Jul 23 11:53:16.184761 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.184367    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b272f586-bd3f-4351-a20c-f44c7d9485e4" containerName="maas-api"
Jul 23 11:53:16.189359 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.189336    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.192145 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.192121    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-839b3c-openshift-default-dockercfg-jhjt6\""
Jul 23 11:53:16.199991 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.199960    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 11:53:16.266281 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqlw\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266452 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266647 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266647 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266647 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.266775 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.266659    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368236 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368411 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368411 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368411 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368411 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368634 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368634 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368634 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqlw\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368634 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368839 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368839 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.368839 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.368831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.369169 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.369143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.369288 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.369273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.371296 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.371269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.371388 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.371321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.377410 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.377386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqlw\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.377508 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.377419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token\") pod \"e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") " pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.501191 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.501088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:16.640703 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.640678    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 11:53:16.642737 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:53:16.642705    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33292aa_0a8a_4628_9f3c_6795923d6674.slice/crio-5193b564198f089df60816e2d8b66eaeb9c42ed99353ff16f5403de2b21a25f6 WatchSource:0}: Error finding container 5193b564198f089df60816e2d8b66eaeb9c42ed99353ff16f5403de2b21a25f6: Status 404 returned error can't find the container with id 5193b564198f089df60816e2d8b66eaeb9c42ed99353ff16f5403de2b21a25f6
Jul 23 11:53:16.644939 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.644909    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:16.645006 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.644974    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:16.645048 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.645006    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:16.879171 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.879056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerStarted","Data":"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"}
Jul 23 11:53:16.879171 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.879138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerStarted","Data":"5193b564198f089df60816e2d8b66eaeb9c42ed99353ff16f5403de2b21a25f6"}
Jul 23 11:53:16.901284 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:16.901228    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podStartSLOduration=0.901206665 podStartE2EDuration="901.206665ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:53:16.899545452 +0000 UTC m=+2570.732353685" watchObservedRunningTime="2026-07-23 11:53:16.901206665 +0000 UTC m=+2570.734014899"
Jul 23 11:53:17.501717 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:17.501677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:17.507453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:17.507428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:17.884205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:17.884121    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:17.885167 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:17.885146    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:18.909453 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.909419    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:53:18.909862 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.909788    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71273b2-6bfe-4165-862c-e846cdede6c7" containerName="istio-proxy"
Jul 23 11:53:18.912794 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.912778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:18.915466 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.915444    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-839b3c-serving-cert\""
Jul 23 11:53:18.916689 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.916670    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:53:18.916790 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.916712    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:53:18.916790 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.916715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7gxng\""
Jul 23 11:53:18.922598 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.922573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:53:18.990185 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.990152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:18.990357 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.990200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:18.990422 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:18.990357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645zf\" (UniqueName: \"kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.091346 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.091317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.091506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.091357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.091506 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.091402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645zf\" (UniqueName: \"kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.091924 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.091902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.093914 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.093892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.100123 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.100074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645zf\" (UniqueName: \"kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf\") pod \"maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.223656 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.223622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.358085 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.358039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:53:19.364966 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:53:19.364935    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbde6c0_0c22_45b3_89bb_8b96c73ba03c.slice/crio-94c8b10d5d81b4810ff6db6b5ac2fc415583d39a3d8e451f86ecdc758c14d1b3 WatchSource:0}: Error finding container 94c8b10d5d81b4810ff6db6b5ac2fc415583d39a3d8e451f86ecdc758c14d1b3: Status 404 returned error can't find the container with id 94c8b10d5d81b4810ff6db6b5ac2fc415583d39a3d8e451f86ecdc758c14d1b3
Jul 23 11:53:19.893901 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.893864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" event={"ID":"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c","Type":"ContainerStarted","Data":"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"}
Jul 23 11:53:19.894134 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.894058    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:19.894134 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.894108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" event={"ID":"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c","Type":"ContainerStarted","Data":"94c8b10d5d81b4810ff6db6b5ac2fc415583d39a3d8e451f86ecdc758c14d1b3"}
Jul 23 11:53:19.919196 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:19.919139    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" podStartSLOduration=1.5104646320000001 podStartE2EDuration="1.91912103s" totalImagesPullingTime="408.656398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:53:19.918802955 +0000 UTC m=+2573.751611189" watchObservedRunningTime="2026-07-23 11:53:19.91912103 +0000 UTC m=+2573.751929264"
Jul 23 11:53:24.404130 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:24.404073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:53:25.904512 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:25.904484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:53:27.294205 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.294174    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 11:53:27.298847 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.298823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.301373 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.301350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c713d8-serving-cert\""
Jul 23 11:53:27.309248 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.309223    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 11:53:27.372635 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.372592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.372635 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.372636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.372862 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.372667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lmx\" (UniqueName: \"kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.473800 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.473760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.473977 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.473807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.473977 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.473854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lmx\" (UniqueName: \"kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.474476 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.474451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.476490 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.476469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.498636 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.498611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lmx\" (UniqueName: \"kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx\") pod \"maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.610245 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.610149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:27.795157 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.795130    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 11:53:27.797917 ip-10-0-129-186 kubenswrapper[2583]: W0723 11:53:27.797893    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec120c9_3d1b_4d35_88c2_5815e9968c73.slice/crio-1071040d42f9b391b6ada606932562d9c5755d5a15d1a0f3ee0ab55ff138a5b6 WatchSource:0}: Error finding container 1071040d42f9b391b6ada606932562d9c5755d5a15d1a0f3ee0ab55ff138a5b6: Status 404 returned error can't find the container with id 1071040d42f9b391b6ada606932562d9c5755d5a15d1a0f3ee0ab55ff138a5b6
Jul 23 11:53:27.931268 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:27.931227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" event={"ID":"1ec120c9-3d1b-4d35-88c2-5815e9968c73","Type":"ContainerStarted","Data":"1071040d42f9b391b6ada606932562d9c5755d5a15d1a0f3ee0ab55ff138a5b6"}
Jul 23 11:53:28.939146 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:28.939107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" event={"ID":"1ec120c9-3d1b-4d35-88c2-5815e9968c73","Type":"ContainerStarted","Data":"d4f1b4f905e7016aaefe785d9025a91e2c30700ca1b343e28d70d477f4002719"}
Jul 23 11:53:28.940479 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:28.940456    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:28.960408 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:28.960353    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" podStartSLOduration=1.585046705 podStartE2EDuration="1.960335834s" totalImagesPullingTime="375.289129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:53:28.959788704 +0000 UTC m=+2582.792596982" watchObservedRunningTime="2026-07-23 11:53:28.960335834 +0000 UTC m=+2582.793144068"
Jul 23 11:53:31.953111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.953062    2583 generic.go:356] "Generic (PLEG): container finished" podID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerID="75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5" exitCode=0
Jul 23 11:53:31.953111 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.953109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerDied","Data":"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"}
Jul 23 11:53:31.953497 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.953470    2583 scope.go:117] "RemoveContainer" containerID="75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"
Jul 23 11:53:31.954441 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.954413    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:31.954492 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.954479    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:31.954531 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:31.954508    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 23 11:53:32.885035 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:32.884994    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:32.959380 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:32.959343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerStarted","Data":"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"}
Jul 23 11:53:33.097713 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:33.097675    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:53:33.501620 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:33.501584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:33.502738 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:33.502717    2583 patch_prober.go:28] interesting pod/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 11:53:33.502796 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:33.502765    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 11:53:34.501797 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:34.501757    2583 patch_prober.go:28] interesting pod/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 11:53:34.502273 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:34.501822    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 11:53:35.502407 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:35.502369    2583 patch_prober.go:28] interesting pod/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 11:53:35.502892 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:35.502435    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 11:53:35.953934 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:35.953905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 11:53:36.502269 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:36.502234    2583 patch_prober.go:28] interesting pod/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 11:53:36.502451 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:36.502290    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 11:53:37.505022 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:37.504994    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:37.505405 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:37.505352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:37.506038 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:37.506020    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:53:59.890146 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:53:59.890055    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:54:25.713602 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:54:25.713563    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:54:39.297699 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:54:39.297666    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:54:57.791209 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:54:57.791170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:55:16.193668 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:55:16.193630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:55:33.909968 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:55:33.909888    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:55:50.494977 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:55:50.494941    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:56:08.100720 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:56:08.100677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:56:26.899000 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:56:26.898968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:56:41.115326 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:56:41.115263    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:57:05.713923 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:57:05.713838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:57:21.197325 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:57:21.197288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:57:43.582832 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:57:43.582796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:57:59.592549 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:57:59.592511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:58:32.812889 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:58:32.812852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:58:46.487417 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:58:46.487382    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:59:04.986403 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:04.986357    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:59:20.281992 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:20.281954    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:59:43.274719 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.274679    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:59:43.275531 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.275472    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" podUID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" containerName="maas-api" containerID="cri-o://58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4" gracePeriod=30
Jul 23 11:59:43.526956 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.526887    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:59:43.562591 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.562554    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" containerID="58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4" exitCode=0
Jul 23 11:59:43.562773 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.562599    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"
Jul 23 11:59:43.562773 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.562607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" event={"ID":"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c","Type":"ContainerDied","Data":"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"}
Jul 23 11:59:43.562773 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.562665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss" event={"ID":"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c","Type":"ContainerDied","Data":"94c8b10d5d81b4810ff6db6b5ac2fc415583d39a3d8e451f86ecdc758c14d1b3"}
Jul 23 11:59:43.562773 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.562691    2583 scope.go:117] "RemoveContainer" containerID="58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"
Jul 23 11:59:43.571588 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.571570    2583 scope.go:117] "RemoveContainer" containerID="58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"
Jul 23 11:59:43.571901 ip-10-0-129-186 kubenswrapper[2583]: E0723 11:59:43.571875    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4\": container with ID starting with 58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4 not found: ID does not exist" containerID="58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"
Jul 23 11:59:43.571999 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.571906    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4"} err="failed to get container status \"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4\": rpc error: code = NotFound desc = could not find container \"58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4\": container with ID starting with 58e0c8ca994d6f5b5ab75a07c80429daf94654fcf48cd6fade474f47d5ad51d4 not found: ID does not exist"
Jul 23 11:59:43.721929 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.721896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-645zf\" (UniqueName: \"kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf\") pod \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") "
Jul 23 11:59:43.722136 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.721954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca\") pod \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") "
Jul 23 11:59:43.722136 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.721978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls\") pod \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\" (UID: \"6fbde6c0-0c22-45b3-89bb-8b96c73ba03c\") "
Jul 23 11:59:43.722361 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.722337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" (UID: "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:59:43.724313 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.724282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" (UID: "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:59:43.724427 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.724405    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf" (OuterVolumeSpecName: "kube-api-access-645zf") pod "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" (UID: "6fbde6c0-0c22-45b3-89bb-8b96c73ba03c"). InnerVolumeSpecName "kube-api-access-645zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:59:43.823685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.823587    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-645zf\" (UniqueName: \"kubernetes.io/projected/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-kube-api-access-645zf\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:59:43.823685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.823632    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:59:43.823685 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.823642    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:59:43.904233 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.904199    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:59:43.915944 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:43.915911    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-839b3c-59dbfd586-pbbss"]
Jul 23 11:59:44.744574 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:44.744537    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" path="/var/lib/kubelet/pods/6fbde6c0-0c22-45b3-89bb-8b96c73ba03c/volumes"
Jul 23 11:59:46.504855 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:46.504823    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 11:59:54.592556 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:54.592480    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 11:59:54.592978 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:54.592709    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy" containerID="cri-o://3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808" gracePeriod=30
Jul 23 11:59:59.854244 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.854181    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 11:59:59.952649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cqlw\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952649 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.952894 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952869    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.953212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.952912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token\") pod \"f33292aa-0a8a-4628-9f3c-6795923d6674\" (UID: \"f33292aa-0a8a-4628-9f3c-6795923d6674\") "
Jul 23 11:59:59.953212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:59:59.953212 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:59:59.953379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953216    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:59:59.953379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953296    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:59:59.953379 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953321    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-workload-certs\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 11:59:59.953529 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:59:59.953578 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.953561    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data" (OuterVolumeSpecName: "istio-data") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:59:59.954966 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.954941    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw" (OuterVolumeSpecName: "kube-api-access-5cqlw") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "kube-api-access-5cqlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:59:59.955288 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.955262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:59:59.955431 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.955411    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token" (OuterVolumeSpecName: "istio-token") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:59:59.955500 ip-10-0-129-186 kubenswrapper[2583]: I0723 11:59:59.955444    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f33292aa-0a8a-4628-9f3c-6795923d6674" (UID: "f33292aa-0a8a-4628-9f3c-6795923d6674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 12:00:00.054564 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054531    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33292aa-0a8a-4628-9f3c-6795923d6674-istiod-ca-cert\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054564 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054561    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-envoy\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054564 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054570    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-podinfo\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054814 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054578    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-token\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054814 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054587    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqlw\" (UniqueName: \"kubernetes.io/projected/f33292aa-0a8a-4628-9f3c-6795923d6674-kube-api-access-5cqlw\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054814 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054596    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-credential-socket\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.054814 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.054604    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33292aa-0a8a-4628-9f3c-6795923d6674-istio-data\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:00.145871 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.145785    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"]
Jul 23 12:00:00.146164 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146151    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" containerName="maas-api"
Jul 23 12:00:00.146217 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146166    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" containerName="maas-api"
Jul 23 12:00:00.146217 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146175    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.146217 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146181    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.146323 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.146323 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146251    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.146323 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.146261    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbde6c0-0c22-45b3-89bb-8b96c73ba03c" containerName="maas-api"
Jul 23 12:00:00.149401 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.149384    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"]
Jul 23 12:00:00.149526 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.149507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"
Jul 23 12:00:00.149634 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.149624    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.149666 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.149636    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerName="istio-proxy"
Jul 23 12:00:00.152419 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.152398    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wx4jb\""
Jul 23 12:00:00.152529 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.152426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"
Jul 23 12:00:00.155794 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.155767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"]
Jul 23 12:00:00.159559 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.159499    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"]
Jul 23 12:00:00.164785 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.164709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"
Jul 23 12:00:00.171177 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.171159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"
Jul 23 12:00:00.344963 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.344815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"]
Jul 23 12:00:00.347671 ip-10-0-129-186 kubenswrapper[2583]: W0723 12:00:00.347643    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4214303_334f_4de3_9205_089a8dfe14b6.slice/crio-9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7 WatchSource:0}: Error finding container 9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7: Status 404 returned error can't find the container with id 9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7
Jul 23 12:00:00.520008 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.519975    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"]
Jul 23 12:00:00.523666 ip-10-0-129-186 kubenswrapper[2583]: W0723 12:00:00.523636    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2106925_a642_47d8_b7d1_0336a89f9611.slice/crio-5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396 WatchSource:0}: Error finding container 5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396: Status 404 returned error can't find the container with id 5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396
Jul 23 12:00:00.644324 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.644289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt" event={"ID":"c2106925-a642-47d8-b7d1-0336a89f9611","Type":"ContainerStarted","Data":"5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396"}
Jul 23 12:00:00.645702 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.645675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp" event={"ID":"c4214303-334f-4de3-9205-089a8dfe14b6","Type":"ContainerStarted","Data":"c24fd93aff44b1ee7a28c0f51a696e1855513cf383c7063938f511d21339188b"}
Jul 23 12:00:00.645842 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.645725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp" event={"ID":"c4214303-334f-4de3-9205-089a8dfe14b6","Type":"ContainerStarted","Data":"9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7"}
Jul 23 12:00:00.647388 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.647361    2583 generic.go:356] "Generic (PLEG): container finished" podID="f33292aa-0a8a-4628-9f3c-6795923d6674" containerID="3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808" exitCode=0
Jul 23 12:00:00.647476 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.647397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerDied","Data":"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"}
Jul 23 12:00:00.647476 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.647463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz" event={"ID":"f33292aa-0a8a-4628-9f3c-6795923d6674","Type":"ContainerDied","Data":"5193b564198f089df60816e2d8b66eaeb9c42ed99353ff16f5403de2b21a25f6"}
Jul 23 12:00:00.647563 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.647414    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"
Jul 23 12:00:00.647563 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.647488    2583 scope.go:117] "RemoveContainer" containerID="3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"
Jul 23 12:00:00.656522 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.656500    2583 scope.go:117] "RemoveContainer" containerID="75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"
Jul 23 12:00:00.662776 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.662730    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp" podStartSLOduration=0.662716007 podStartE2EDuration="662.716007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:00:00.661557159 +0000 UTC m=+2974.494365411" watchObservedRunningTime="2026-07-23 12:00:00.662716007 +0000 UTC m=+2974.495524240"
Jul 23 12:00:00.665766 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.665749    2583 scope.go:117] "RemoveContainer" containerID="3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"
Jul 23 12:00:00.666031 ip-10-0-129-186 kubenswrapper[2583]: E0723 12:00:00.666012    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808\": container with ID starting with 3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808 not found: ID does not exist" containerID="3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"
Jul 23 12:00:00.666109 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.666040    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808"} err="failed to get container status \"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808\": rpc error: code = NotFound desc = could not find container \"3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808\": container with ID starting with 3e77c1b799b2d36cc2fc90e226c8f56cc0f2ce4f8ccc81032e5decacc7ea2808 not found: ID does not exist"
Jul 23 12:00:00.666109 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.666057    2583 scope.go:117] "RemoveContainer" containerID="75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"
Jul 23 12:00:00.666305 ip-10-0-129-186 kubenswrapper[2583]: E0723 12:00:00.666290    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5\": container with ID starting with 75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5 not found: ID does not exist" containerID="75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"
Jul 23 12:00:00.666345 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.666311    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5"} err="failed to get container status \"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5\": rpc error: code = NotFound desc = could not find container \"75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5\": container with ID starting with 75180c5f64feff1a6a2cc2367702444f67476100c116599f1cba74336cb30ad5 not found: ID does not exist"
Jul 23 12:00:00.676040 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.676015    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 12:00:00.679279 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.679255    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-839b3c-openshift-default-79597fbbc-xcvkz"]
Jul 23 12:00:00.745667 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:00.745633    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f33292aa-0a8a-4628-9f3c-6795923d6674" path="/var/lib/kubelet/pods/f33292aa-0a8a-4628-9f3c-6795923d6674/volumes"
Jul 23 12:00:01.653484 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:01.653392    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2106925-a642-47d8-b7d1-0336a89f9611" containerID="bd6f319190982318441f2fcb73560440b09a6408e0d58d63fe62429d8bbc5a42" exitCode=0
Jul 23 12:00:01.653904 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:01.653485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt" event={"ID":"c2106925-a642-47d8-b7d1-0336a89f9611","Type":"ContainerDied","Data":"bd6f319190982318441f2fcb73560440b09a6408e0d58d63fe62429d8bbc5a42"}
Jul 23 12:00:01.654911 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:01.654874    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4214303-334f-4de3-9205-089a8dfe14b6" containerID="c24fd93aff44b1ee7a28c0f51a696e1855513cf383c7063938f511d21339188b" exitCode=0
Jul 23 12:00:01.655030 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:01.654958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp" event={"ID":"c4214303-334f-4de3-9205-089a8dfe14b6","Type":"ContainerDied","Data":"c24fd93aff44b1ee7a28c0f51a696e1855513cf383c7063938f511d21339188b"}
Jul 23 12:00:02.797056 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:02.797034    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"
Jul 23 12:00:02.831870 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:02.831843    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"
Jul 23 12:00:03.665373 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.665336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"
Jul 23 12:00:03.665576 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.665340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt" event={"ID":"c2106925-a642-47d8-b7d1-0336a89f9611","Type":"ContainerDied","Data":"5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396"}
Jul 23 12:00:03.665576 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.665509    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f1802eb84d732962a356f1aa1248b2e76ddcc5e051eed3c2c98860bddcd8396"
Jul 23 12:00:03.666750 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.666732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp"
Jul 23 12:00:03.666750 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.666738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-7dlxp" event={"ID":"c4214303-334f-4de3-9205-089a8dfe14b6","Type":"ContainerDied","Data":"9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7"}
Jul 23 12:00:03.666898 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:03.666778    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1071be39f2a70fcf46612be5d6365ace24777c68d3e82ad09f5f39682e87b7"
Jul 23 12:00:05.545542 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.545506    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 12:00:05.546001 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.545762    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" podUID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" containerName="maas-api" containerID="cri-o://d4f1b4f905e7016aaefe785d9025a91e2c30700ca1b343e28d70d477f4002719" gracePeriod=30
Jul 23 12:00:05.573727 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.573702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"]
Jul 23 12:00:05.576626 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.576603    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c713d8-29746800-jlptt"]
Jul 23 12:00:05.679043 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.679010    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" containerID="d4f1b4f905e7016aaefe785d9025a91e2c30700ca1b343e28d70d477f4002719" exitCode=0
Jul 23 12:00:05.679237 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.679085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" event={"ID":"1ec120c9-3d1b-4d35-88c2-5815e9968c73","Type":"ContainerDied","Data":"d4f1b4f905e7016aaefe785d9025a91e2c30700ca1b343e28d70d477f4002719"}
Jul 23 12:00:05.796022 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.795962    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 12:00:05.906702 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.906671    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca\") pod \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") "
Jul 23 12:00:05.906702 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.906716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5lmx\" (UniqueName: \"kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx\") pod \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") "
Jul 23 12:00:05.906980 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.906748    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls\") pod \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\" (UID: \"1ec120c9-3d1b-4d35-88c2-5815e9968c73\") "
Jul 23 12:00:05.907115 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.907067    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ec120c9-3d1b-4d35-88c2-5815e9968c73" (UID: "1ec120c9-3d1b-4d35-88c2-5815e9968c73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:00:05.909034 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.909001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ec120c9-3d1b-4d35-88c2-5815e9968c73" (UID: "1ec120c9-3d1b-4d35-88c2-5815e9968c73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:00:05.909155 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:05.909060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx" (OuterVolumeSpecName: "kube-api-access-v5lmx") pod "1ec120c9-3d1b-4d35-88c2-5815e9968c73" (UID: "1ec120c9-3d1b-4d35-88c2-5815e9968c73"). InnerVolumeSpecName "kube-api-access-v5lmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:00:06.007803 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.007752    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec120c9-3d1b-4d35-88c2-5815e9968c73-openshift-service-ca\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.007803 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.007797    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5lmx\" (UniqueName: \"kubernetes.io/projected/1ec120c9-3d1b-4d35-88c2-5815e9968c73-kube-api-access-v5lmx\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.007803 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.007808    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ec120c9-3d1b-4d35-88c2-5815e9968c73-maas-api-tls\") on node \"ip-10-0-129-186.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.684970 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.684942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"
Jul 23 12:00:06.684970 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.684951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj" event={"ID":"1ec120c9-3d1b-4d35-88c2-5815e9968c73","Type":"ContainerDied","Data":"1071040d42f9b391b6ada606932562d9c5755d5a15d1a0f3ee0ab55ff138a5b6"}
Jul 23 12:00:06.685474 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.685004    2583 scope.go:117] "RemoveContainer" containerID="d4f1b4f905e7016aaefe785d9025a91e2c30700ca1b343e28d70d477f4002719"
Jul 23 12:00:06.706779 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.706754    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 12:00:06.710483 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.710459    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c713d8-7c86bcb75-bn6rj"]
Jul 23 12:00:06.749975 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.749943    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" path="/var/lib/kubelet/pods/1ec120c9-3d1b-4d35-88c2-5815e9968c73/volumes"
Jul 23 12:00:06.750244 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:06.750230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2106925-a642-47d8-b7d1-0336a89f9611" path="/var/lib/kubelet/pods/c2106925-a642-47d8-b7d1-0336a89f9611/volumes"
Jul 23 12:00:08.287602 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:08.287565    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l76pr"]
Jul 23 12:00:23.925883 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:23.925856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684564f56d-p9rdw_06300105-ae5b-47a7-a6c6-1b99373abb49/copy-manifests/0.log"
Jul 23 12:00:23.934967 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:23.934933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684564f56d-p9rdw_06300105-ae5b-47a7-a6c6-1b99373abb49/manager/0.log"
Jul 23 12:00:24.490687 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:24.490648    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746785-7q48q_b1ff0c31-956c-4da9-a65e-f42f511bd48a/cleanup/0.log"
Jul 23 12:00:24.610943 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:24.610903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746800-7dlxp_c4214303-334f-4de3-9205-089a8dfe14b6/cleanup/0.log"
Jul 23 12:00:26.754148 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:26.754109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l76pr_01191d10-8c6a-4454-a048-ebc1b9cd3593/limitador/0.log"
Jul 23 12:00:27.230344 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:27.230305    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vnwg6_6d778b67-9a4c-4b03-8e5d-ad26da471f71/istio-proxy/0.log"
Jul 23 12:00:27.355999 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:27.355963    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wjfns_9082d1fd-0256-4f1f-8a97-01580c15e3aa/discovery/0.log"
Jul 23 12:00:28.608191 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:28.608158    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ql4hh_714c15a2-d7b6-40d8-8735-4328888d524f/main/0.log"
Jul 23 12:00:37.296787 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:37.296750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mts9v_70cb2736-a420-4e4d-be73-a9a12d0c3f15/global-pull-secret-syncer/0.log"
Jul 23 12:00:37.410510 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:37.410482    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7nbsm_498c5f37-ded0-46f2-8784-4598551995eb/konnectivity-agent/0.log"
Jul 23 12:00:37.433187 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:37.433162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-186.ec2.internal_d5ee178cb91c3b2f8f69b5e375b061e5/haproxy/0.log"
Jul 23 12:00:42.920559 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:42.920529    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l76pr_01191d10-8c6a-4454-a048-ebc1b9cd3593/limitador/0.log"
Jul 23 12:00:45.183844 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.183800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-792pm_f3e17a55-6ab8-4f46-a232-56b3d5ef7c10/node-exporter/0.log"
Jul 23 12:00:45.207249 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.207221    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-792pm_f3e17a55-6ab8-4f46-a232-56b3d5ef7c10/kube-rbac-proxy/0.log"
Jul 23 12:00:45.237054 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.237029    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-792pm_f3e17a55-6ab8-4f46-a232-56b3d5ef7c10/init-textfile/0.log"
Jul 23 12:00:45.503077 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.503002    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/prometheus/0.log"
Jul 23 12:00:45.523482 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.523456    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/config-reloader/0.log"
Jul 23 12:00:45.545811 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.545787    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/thanos-sidecar/0.log"
Jul 23 12:00:45.570462 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.570439    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/kube-rbac-proxy-web/0.log"
Jul 23 12:00:45.596841 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.596821    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/kube-rbac-proxy/0.log"
Jul 23 12:00:45.627349 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.627323    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/kube-rbac-proxy-thanos/0.log"
Jul 23 12:00:45.652798 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:45.652777    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3d955b53-25f7-4383-959a-0c1b36108937/init-config-reloader/0.log"
Jul 23 12:00:47.915425 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915394    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"]
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915668    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2106925-a642-47d8-b7d1-0336a89f9611" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915678    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2106925-a642-47d8-b7d1-0336a89f9611" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915697    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" containerName="maas-api"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915703    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" containerName="maas-api"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915712    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4214303-334f-4de3-9205-089a8dfe14b6" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915717    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4214303-334f-4de3-9205-089a8dfe14b6" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915765    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4214303-334f-4de3-9205-089a8dfe14b6" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2106925-a642-47d8-b7d1-0336a89f9611" containerName="cleanup"
Jul 23 12:00:47.915813 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.915778    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec120c9-3d1b-4d35-88c2-5815e9968c73" containerName="maas-api"
Jul 23 12:00:47.918865 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.918839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:47.924652 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.924630    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8hbw\"/\"openshift-service-ca.crt\""
Jul 23 12:00:47.924791 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.924725    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8hbw\"/\"default-dockercfg-5ql9l\""
Jul 23 12:00:47.925861 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.925843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8hbw\"/\"kube-root-ca.crt\""
Jul 23 12:00:47.941835 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.941803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"]
Jul 23 12:00:47.958681 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.958655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-podres\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:47.958809 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.958693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-proc\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:47.958809 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.958717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-sys\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:47.958809 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.958782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmg84\" (UniqueName: \"kubernetes.io/projected/4fbaf168-39e2-4d9f-aa94-6986df4e528c-kube-api-access-nmg84\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:47.958933 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:47.958838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-lib-modules\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059249 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-lib-modules\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-podres\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-proc\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-sys\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-sys\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-lib-modules\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059448 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-proc\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059691 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg84\" (UniqueName: \"kubernetes.io/projected/4fbaf168-39e2-4d9f-aa94-6986df4e528c-kube-api-access-nmg84\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.059691 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.059448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fbaf168-39e2-4d9f-aa94-6986df4e528c-podres\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.070645 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.070603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg84\" (UniqueName: \"kubernetes.io/projected/4fbaf168-39e2-4d9f-aa94-6986df4e528c-kube-api-access-nmg84\") pod \"perf-node-gather-daemonset-tp6vq\" (UID: \"4fbaf168-39e2-4d9f-aa94-6986df4e528c\") " pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.228589 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.228556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.366167 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.366143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"]
Jul 23 12:00:48.368472 ip-10-0-129-186 kubenswrapper[2583]: W0723 12:00:48.368445    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fbaf168_39e2_4d9f_aa94_6986df4e528c.slice/crio-fb9e6021634fb33675df1ca4f806a61cd0635dcd0fb98b1f9bf972d32e7469f9 WatchSource:0}: Error finding container fb9e6021634fb33675df1ca4f806a61cd0635dcd0fb98b1f9bf972d32e7469f9: Status 404 returned error can't find the container with id fb9e6021634fb33675df1ca4f806a61cd0635dcd0fb98b1f9bf972d32e7469f9
Jul 23 12:00:48.863073 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.863039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq" event={"ID":"4fbaf168-39e2-4d9f-aa94-6986df4e528c","Type":"ContainerStarted","Data":"c13c732ef76c81073fd0cee919dc8028ad3b29d108b0d7fb4e90707c03ebadb3"}
Jul 23 12:00:48.863276 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.863117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq" event={"ID":"4fbaf168-39e2-4d9f-aa94-6986df4e528c","Type":"ContainerStarted","Data":"fb9e6021634fb33675df1ca4f806a61cd0635dcd0fb98b1f9bf972d32e7469f9"}
Jul 23 12:00:48.863276 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.863147    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:48.879853 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:48.879803    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq" podStartSLOduration=1.8797883290000001 podStartE2EDuration="1.879788329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:00:48.87860627 +0000 UTC m=+3022.711414507" watchObservedRunningTime="2026-07-23 12:00:48.879788329 +0000 UTC m=+3022.712596562"
Jul 23 12:00:50.154735 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:50.154702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9ddn_4548a142-564b-4bd2-8c29-5218892673ea/dns/0.log"
Jul 23 12:00:50.184688 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:50.184660    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9ddn_4548a142-564b-4bd2-8c29-5218892673ea/kube-rbac-proxy/0.log"
Jul 23 12:00:50.282018 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:50.281990    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tvvkn_e655262a-c02f-4320-b3c9-6b4c22b446b0/dns-node-resolver/0.log"
Jul 23 12:00:50.881661 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:50.881630    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shvrq_1075af97-b1ff-40bc-8742-d35953d18a85/node-ca/0.log"
Jul 23 12:00:51.895839 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:51.895786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vnwg6_6d778b67-9a4c-4b03-8e5d-ad26da471f71/istio-proxy/0.log"
Jul 23 12:00:52.027535 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:52.027501    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wjfns_9082d1fd-0256-4f1f-8a97-01580c15e3aa/discovery/0.log"
Jul 23 12:00:52.992429 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:52.992400    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tqzkh_fd804822-80f3-4c89-8b6c-ca87eff98f8c/serve-healthcheck-canary/0.log"
Jul 23 12:00:53.711403 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:53.711373    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmfrz_4b021505-da25-4104-a92e-a9ca5348ac58/kube-rbac-proxy/0.log"
Jul 23 12:00:53.750999 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:53.750969    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmfrz_4b021505-da25-4104-a92e-a9ca5348ac58/exporter/0.log"
Jul 23 12:00:53.777433 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:53.777404    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmfrz_4b021505-da25-4104-a92e-a9ca5348ac58/extractor/0.log"
Jul 23 12:00:54.877293 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:54.877262    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8hbw/perf-node-gather-daemonset-tp6vq"
Jul 23 12:00:56.557413 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:56.557386    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684564f56d-p9rdw_06300105-ae5b-47a7-a6c6-1b99373abb49/manager/0.log"
Jul 23 12:00:56.578883 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:56.578856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-684564f56d-p9rdw_06300105-ae5b-47a7-a6c6-1b99373abb49/copy-manifests/0.log"
Jul 23 12:00:58.003410 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:58.003381    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57b4b866cd-pzmkk_12aadc1e-4ae6-4639-b9ed-ff4e5dfe5890/manager/0.log"
Jul 23 12:00:58.204787 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:58.204753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vnwg6_6d778b67-9a4c-4b03-8e5d-ad26da471f71/istio-proxy/0.log"
Jul 23 12:00:58.320841 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:58.320788    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wjfns_9082d1fd-0256-4f1f-8a97-01580c15e3aa/discovery/0.log"
Jul 23 12:00:59.694937 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:59.694905    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vnwg6_6d778b67-9a4c-4b03-8e5d-ad26da471f71/istio-proxy/0.log"
Jul 23 12:00:59.824767 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:00:59.824713    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wjfns_9082d1fd-0256-4f1f-8a97-01580c15e3aa/discovery/0.log"
Jul 23 12:01:04.687651 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.687612    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4jcc8_dd73de0a-9361-4d9e-9486-a4beb65010c9/kube-multus/0.log"
Jul 23 12:01:04.740042 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.740011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/kube-multus-additional-cni-plugins/0.log"
Jul 23 12:01:04.758217 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.758192    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/egress-router-binary-copy/0.log"
Jul 23 12:01:04.776824 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.776799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/cni-plugins/0.log"
Jul 23 12:01:04.796027 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.796000    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/bond-cni-plugin/0.log"
Jul 23 12:01:04.814802 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.814779    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/routeoverride-cni/0.log"
Jul 23 12:01:04.836148 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.836124    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/whereabouts-cni-bincopy/0.log"
Jul 23 12:01:04.855174 ip-10-0-129-186 kubenswrapper[2583]: I0723 12:01:04.855149    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5p4hs_c43c0bf8-f3c5-4b05-8800-c5a7a186703c/whereabouts-cni/0.log"